pip install ydata-quality

# create the main class that holds all quality modulesdq = DataQuality(df=df)
# run the testsresults = dq.evaluate()

Warnings:  TOTAL: 5 warning(s)  Priority 1: 1 warning(s)  Priority 2: 4 warning(s)
Priority 1 - heavy impact expected:  * [DUPLICATES - DUPLICATE COLUMNS] Found 1 columns with exactly the same feature values as other columns.Priority 2 - usage allowed, limited human intelligibility:  * [DATA RELATIONS - HIGH COLLINEARITY - NUMERICAL] Found 3 numerical variables with high Variance Inflation Factor (VIF>5.0). The variables listed in results are highly collinear with other variables in the dataset. These will make model explainability harder and potentially give way to issues like overfitting. Depending on your end goal you might want to remove the highest VIF variables.  * [ERRONEOUS DATA - PREDEFINED ERRONEOUS DATA] Found 1960 ED values in the dataset.  * [DATA RELATIONS - HIGH COLLINEARITY - CATEGORICAL] Found 10 categorical variables with significant collinearity (p-value < 0.05). The variables listed in results are highly collinear with other variables in the dataset and sorted descending according to propensity. These will make model explainability harder and potentially give way to issues like overfitting. Depending on your end goal you might want to remove variables following the provided order.  * [DUPLICATES - EXACT DUPLICATES] Found 3 instances with exact duplicate feature values.

dq.get_warnings(test="DuplicateColumns")

[QualityWarning(category='Duplicates', test='Duplicate Columns', description='Found 1 columns with exactly the same feature values as other columns.', priority=<Priority.P1: 1>, data={'workclass': ['workclass2']})]

Warnings:  TOTAL: 2 warning(s)  Priority 2: 2 warning(s)
Priority 2 - usage allowed, limited human intelligibility:  * [BIAS&FAIRNESS - PROXY IDENTIFICATION] Found 1 feature pairs of correlation to sensitive attributes with values higher than defined threshold (0.5).  * [BIAS&FAIRNESS - SENSITIVE ATTRIBUTE REPRESENTATIVITY] Found 2 values of 'race' sensitive attribute with low representativity in the dataset (below 1.00%).

bf.get_warnings(test='Proxy Identification')

def improve_quality(df: pd.DataFrame):    """Clean the data based on the Data Quality issues found previously."""    # Bias & Fairness    df = df.replace({'relationship': {'Husband': 'Married', 'Wife': 'Married'}}) # Substitute gender-based 'Husband'/'Wife' for generic 'Married'
    # Duplicates    df = df.drop(columns=['workclass2']) # Remove the duplicated column    df = df.drop_duplicates()            # Remove exact feature value duplicates
    return df
clean_df = improve_quality(df.copy())

*DataQuality Engine Report:*
Warnings:  TOTAL: 3 warning(s)  Priority 2: 3 warning(s)

Priority 2 - usage allowed, limited human intelligibility:  * [ERRONEOUS DATA - PREDEFINED ERRONEOUS DATA] Found 1360 ED values in the dataset.  * [DATA RELATIONS - HIGH COLLINEARITY - NUMERICAL] Found 3 numerical variables with high Variance Inflation Factor (VIF>5.0). The variables listed in results are highly collinear with other variables in the dataset. These will make model explainability harder and potentially give way to issues like overfitting. Depending on your end goal you might want to remove the highest VIF variables.  * [DATA RELATIONS - HIGH COLLINEARITY - CATEGORICAL] Found 9 categorical variables with significant collinearity (p-value < 0.05). The variables listed in results are highly collinear with other variables in the dataset and sorted descending according to propensity. These will make model explainability harder and potentially give way to issues like overfitting. Depending on your end goal you might want to remove variables following the provided order.
*Bias & Fairness Report:*
Warnings:  TOTAL: 1 warning(s)  Priority 2: 1 warning(s)

Priority 2 - usage allowed, limited human intelligibility:  * [BIAS&FAIRNESS - SENSITIVE ATTRIBUTE REPRESENTATIVITY] Found 2 values of 'race' sensitive attribute with low representativity in the dataset (below 1.00%).