Diagnosing Data Quality and Process Relationships in an Iron Ore Flotation Plant

Setup

The dataset holds six months of operating data from an iron ore flotation plant, recorded at 20-second intervals. It is published on Kaggle as Quality Prediction in a Mining Process. Values use commas as decimal separators, so pandas needs to be told how to parse them, and the date column needs converting from text to a proper timestamp.

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('/work/MiningProcess_Flotation_Plant_Database.csv', decimal=',')
df['date'] = pd.to_datetime(df['date'])

print(df.shape)
df.head()

(737453, 24)

	date	% Iron Feed	% Silica Feed	Starch Flow	Amina Flow	Ore Pulp Flow	Ore Pulp pH	Ore Pulp Density	Flotation Column 01 Air Flow	Flotation Column 02 Air Flow	...	Flotation Column 07 Air Flow	Flotation Column 01 Level	Flotation Column 02 Level	Flotation Column 03 Level	Flotation Column 04 Level	Flotation Column 05 Level	Flotation Column 06 Level	Flotation Column 07 Level	% Iron Concentrate	% Silica Concentrate
0	2017-03-10 01:00:00	55.2	16.98	3019.53	557.434	395.713	10.0664	1.74	249.214	253.235	...	250.884	457.396	432.962	424.954	443.558	502.255	446.370	523.344	66.91	1.31
1	2017-03-10 01:00:00	55.2	16.98	3024.41	563.965	397.383	10.0672	1.74	249.719	250.532	...	248.994	451.891	429.560	432.939	448.086	496.363	445.922	498.075	66.91	1.31
2	2017-03-10 01:00:00	55.2	16.98	3043.46	568.054	399.668	10.0680	1.74	249.741	247.874	...	248.071	451.240	468.927	434.610	449.688	484.411	447.826	458.567	66.91	1.31
3	2017-03-10 01:00:00	55.2	16.98	3047.36	568.665	397.939	10.0689	1.74	249.917	254.487	...	251.147	452.441	458.165	442.865	446.210	471.411	437.690	427.669	66.91	1.31
4	2017-03-10 01:00:00	55.2	16.98	3033.69	558.167	400.254	10.0697	1.74	250.203	252.136	...	248.928	452.441	452.900	450.523	453.670	462.598	443.682	425.679	66.91	1.31

5 rows × 24 columns

Data quality audit

Before analysing anything, it is worth checking whether the data is what it claims to be. The dataset is described as covering March to September 2017, and every column reports a full 737,453 values with no nulls. Both statements are true and both are misleading.

hourly_timestamps = df['date'].drop_duplicates().sort_values()

gaps = hourly_timestamps.diff()

gaps.value_counts()

date
0 days 01:00:00     4095
13 days 07:00:00       1
Name: count, dtype: int64

gaps[gaps > pd.Timedelta('1 hour')]

26814   13 days 07:00:00
Name: date, dtype: timedelta64[ns]

df['date'].nunique()

4097

gap_end = gaps[gaps > pd.Timedelta('1 hour')].index[0]
gap_position = hourly_timestamps.index.get_loc(gap_end)

print('Data stops at:', hourly_timestamps.iloc[gap_position - 1])
print('Data resumes at:', hourly_timestamps.iloc[gap_position])

Data stops at: 2017-03-16 05:00:00
Data resumes at: 2017-03-29 12:00:00

The dataset is described as covering March to September 2017, but it is not continuous. Recording stops on 16 March and does not resume until 29 March. The file contains 4,097 distinct hours where an unbroken six-month record would contain roughly 4,400. No null check would reveal this, because the missing hours are not blank rows. They were never written at all. March is represented by 209 hours across ten calendar days rather than a full month, so any monthly summary of March is based on roughly a third of the period it appears to describe.

Two flotation columns behave differently

air_flow_cols = [c for c in df.columns if 'Air Flow' in c]

df[air_flow_cols].describe().T

	count	mean	std	min	25%	50%	75%	max
Flotation Column 01 Air Flow	737453.0	280.151856	29.621288	175.510	250.281000	299.34400	300.149000	373.871
Flotation Column 02 Air Flow	737453.0	277.159965	30.149357	175.156	250.457000	296.22300	300.690000	375.992
Flotation Column 03 Air Flow	737453.0	281.082397	28.558268	176.469	250.855000	298.69600	300.382000	364.346
Flotation Column 04 Air Flow	737453.0	299.447794	2.572538	292.195	298.262566	299.80500	300.638000	305.871
Flotation Column 05 Air Flow	737453.0	299.917814	3.636579	286.295	298.068000	299.88712	301.791137	310.270
Flotation Column 06 Air Flow	737453.0	292.071485	30.217804	189.928	262.541000	299.47700	303.061000	370.910
Flotation Column 07 Air Flow	737453.0	290.754856	28.670105	185.962	256.302000	299.01100	301.904000	371.593

df[air_flow_cols].std().sort_values()

Flotation Column 04 Air Flow     2.572538
Flotation Column 05 Air Flow     3.636579
Flotation Column 03 Air Flow    28.558268
Flotation Column 07 Air Flow    28.670105
Flotation Column 01 Air Flow    29.621288
Flotation Column 02 Air Flow    30.149357
Flotation Column 06 Air Flow    30.217804
dtype: float64

Five of the seven flotation columns show standard deviations near 29 in air flow, with readings ranging from roughly 175 to 375. Columns 04 and 05 show standard deviations of 2.6 and 3.6 and never fall below 286 across 737,453 readings. Two of the seven therefore carry roughly a tenth of the variation of the others, which limits how much they can explain in any comparison across columns.

Reagent flow at zero

Starch is one of two chemicals dosed into the process. Its average flow is around 2,869, but its recorded minimum is close to zero.

df['Starch Flow'].min()

np.float64(0.00202596)

near_zero_starch = df[df['Starch Flow'] < 100]

print('Rows with starch flow under 100:', len(near_zero_starch))
print('Distinct hours affected:', near_zero_starch['date'].nunique())

near_zero_starch['date'].dt.date.value_counts().sort_index()

Rows with starch flow under 100: 3976
Distinct hours affected: 1111

date
2017-03-10     7
2017-03-11     4
2017-03-12     2
2017-03-13     2
2017-03-14     1
              ..
2017-09-05    49
2017-09-06    14
2017-09-07    15
2017-09-08     6
2017-09-09    10
Name: count, Length: 164, dtype: int64

Starch flow records a minimum of 0.002 against a mean near 2,869. These near-zero readings appear on 164 separate dates across the full six months rather than clustering in any single period. The pattern is consistent with intermittent sensor dropouts or brief low-flow operating periods, though operator and maintenance records would be needed to determine which. The readings account for 0.54% of rows, but they fall within 1,111 of the 4,097 hours in the dataset, so 27% of hourly averages contain at least one. The effect on any individual hourly mean is small, since a typical affected hour contains fewer than four such readings out of 180.

The resolution problem

The dataset mixes two very different measurement rates. Most sensors report every 20 seconds. The two quality figures, iron and silica concentrate, come from laboratory tests taken once an hour. Both sit in the same table with the same timestamp, which makes them look comparable when they are not.

readings_per_hour = df.groupby('date').size()

readings_per_hour.value_counts()

180    4095
174       1
179       1
Name: count, dtype: int64

one_hour = df[df['date'] == '2017-06-01 12:00:00']

print('Rows in this hour:', len(one_hour))
print('Distinct iron concentrate values:', one_hour['% Iron Concentrate'].nunique())
print('Distinct ore pulp pH values:', one_hour['Ore Pulp pH'].nunique())

Rows in this hour: 180
Distinct iron concentrate values: 1
Distinct ore pulp pH values: 180

Almost every hour in the dataset contains exactly 180 readings, one every 20 seconds, with two exceptions at 174 and 179 that fall on the boundaries of the recording gap. Within a single hour, ore pulp pH takes 180 distinct values while iron concentrate takes one, repeated across every row. The quality figures are hourly lab results copied across the hour they belong to. Correlating them against 20-second sensor data therefore compares 24 real measurements per day against roughly 4,300 padded rows. The duplication carries no information, and any correlation computed at this resolution is unreliable. Averaging each sensor to one value per hour puts both sides of the comparison on the same footing.

hourly = df.groupby('date').mean()

print(hourly.shape)
hourly.head()

(4097, 23)

	% Iron Feed	% Silica Feed	Starch Flow	Amina Flow	Ore Pulp Flow	Ore Pulp pH	Ore Pulp Density	Flotation Column 01 Air Flow	Flotation Column 02 Air Flow	Flotation Column 03 Air Flow	...	Flotation Column 07 Air Flow	Flotation Column 01 Level	Flotation Column 02 Level	Flotation Column 03 Level	Flotation Column 04 Level	Flotation Column 05 Level	Flotation Column 06 Level	Flotation Column 07 Level	% Iron Concentrate	% Silica Concentrate
date
2017-03-10 01:00:00	55.2	16.98	3162.625026	578.786678	398.753368	10.113487	1.729558	251.166672	250.226086	250.178287	...	250.208184	450.383776	446.891845	450.474523	449.912259	455.792161	464.383310	450.532747	66.91	1.31
2017-03-10 02:00:00	55.2	16.98	3133.256389	537.219661	399.871822	10.129742	1.667784	249.880589	250.214050	250.033317	...	249.897572	449.373361	450.249356	450.081222	450.328806	448.722983	455.501528	451.387700	67.06	1.11
2017-03-10 03:00:00	55.2	16.98	3479.482944	591.906744	398.763806	10.048403	1.732711	250.161328	250.104167	250.046350	...	250.484183	449.972878	450.868711	450.901822	451.145822	451.134189	459.981311	450.296722	66.97	1.27
2017-03-10 04:00:00	55.2	16.98	3228.036436	593.170106	399.866983	9.918614	1.731056	250.208772	250.204761	250.120861	...	250.157622	487.940706	491.462111	487.387206	494.528183	495.664011	502.763850	494.939889	66.75	1.36
2017-03-10 05:00:00	55.2	16.98	3327.280739	619.710806	399.615089	9.746029	1.765879	249.917800	250.160494	250.013500	...	250.078639	549.031539	549.983156	549.459572	549.975483	549.512533	560.696300	550.271772	66.63	1.34

5 rows × 23 columns

Revisiting the flotation columns

stuck_check = hourly[['Flotation Column 04 Air Flow', 'Flotation Column 05 Air Flow']].nunique()

print(stuck_check)
print()
print('Total hours:', len(hourly))

Flotation Column 04 Air Flow    3571
Flotation Column 05 Air Flow    3605
dtype: int64

Total hours: 4097

hourly[air_flow_cols].nunique()

Flotation Column 01 Air Flow    4032
Flotation Column 02 Air Flow    4067
Flotation Column 03 Air Flow    4000
Flotation Column 04 Air Flow    3571
Flotation Column 05 Air Flow    3605
Flotation Column 06 Air Flow    4055
Flotation Column 07 Air Flow    4059
dtype: int64

With the data aggregated to hourly, columns 04 and 05 produce 3,571 and 3,605 distinct hourly averages across 4,097 hours, against 4,000 to 4,067 for the other five. All seven vary. The difference between them is one of magnitude, not of whether the sensors are working, which rules out a frozen or interpolated channel as the explanation.

Choosing the right target variable

The obvious variable to analyse is iron concentrate, since iron purity is what the plant sells. It is also the variable the data has least to say about.

targets = ['% Iron Concentrate', '% Silica Concentrate']

hourly[targets].describe().T

	count	mean	std	min	25%	50%	75%	max
% Iron Concentrate	4097.0	65.050084	1.118099	62.05	64.37	65.21	65.86	68.01
% Silica Concentrate	4097.0	2.326754	1.124783	0.60	1.44	2.00	3.01	5.53

for col in targets:
    spread = hourly[col].std() / hourly[col].mean() * 100
    print(f'{col}: mean {hourly[col].mean():.2f}, std {hourly[col].std():.2f}, relative spread {spread:.1f}%')

% Iron Concentrate: mean 65.05, std 1.12, relative spread 1.7%
% Silica Concentrate: mean 2.33, std 1.12, relative spread 48.3%

hourly['% Iron Concentrate'].corr(hourly['% Silica Concentrate'])

np.float64(-0.8012080236391909)

Iron and silica concentrate move by almost exactly the same absolute amount, with standard deviations of 1.118 and 1.125. Relative to their means they are nothing alike. Iron averages 65.05 and varies by 1.7% around that figure; silica averages 2.33 and varies by 48.3%. Iron is tightly controlled and holds close to its target, which is what a well-run plant should produce. Because it shows so little proportional variation, silica offers a more sensitive target for studying changes in process performance. The correlation between the two is -0.80, explaining about 64% of their shared linear variation and leaving substantial movement in silica that a simple linear relationship with iron does not capture. The analysis that follows uses silica concentrate as the target.

fig, axes = plt.subplots(1, 2, figsize=(12, 4), sharey=True)

for ax, col, colour in zip(axes, targets, ['tab:blue', 'darkorange']):
    pct = (hourly[col] / hourly[col].mean() - 1) * 100
    pct.plot(ax=ax, linewidth=0.5, color=colour)
    ax.set_title(col)
    ax.set_ylabel('% deviation from mean')
    ax.axhline(0, color='grey', linewidth=0.8)

plt.tight_layout()
plt.show()

Both variables plotted as percentage deviation from their own mean, on a shared axis. Iron concentrate stays within a few percent of its average for six months. Silica concentrate routinely swings more than 100% above and 50% below. The straight diagonal in mid-March is the thirteen-day recording gap, which matplotlib draws as a connecting line rather than a break.

What moves the impurity

With the data at a consistent hourly resolution and silica concentrate as the target, the question becomes which plant conditions move alongside it.

silica_corr = hourly.corr()['% Silica Concentrate'].drop('% Silica Concentrate').sort_values()

silica_corr

% Iron Concentrate             -0.801208
Flotation Column 01 Air Flow   -0.220869
Flotation Column 03 Air Flow   -0.220480
Flotation Column 05 Level      -0.190820
Flotation Column 04 Level      -0.179396
Flotation Column 02 Air Flow   -0.171573
Flotation Column 07 Level      -0.165589
Ore Pulp pH                    -0.151487
Flotation Column 06 Level      -0.122022
Starch Flow                    -0.084729
% Iron Feed                    -0.077112
Flotation Column 07 Air Flow   -0.075650
Flotation Column 06 Air Flow   -0.050716
Flotation Column 05 Air Flow   -0.010148
Flotation Column 04 Air Flow   -0.005289
Ore Pulp Flow                   0.009668
Flotation Column 03 Level       0.015847
Flotation Column 01 Level       0.018310
Flotation Column 02 Level       0.034116
Ore Pulp Density                0.050954
% Silica Feed                   0.072780
Amina Flow                      0.170938
Name: % Silica Concentrate, dtype: float64

plt.figure(figsize=(8, 7))

silica_corr.plot(kind='barh', color=['tab:blue' if v < 0 else 'darkorange' for v in silica_corr])

plt.axvline(0, color='grey', linewidth=0.8)
plt.title('Correlation with % Silica Concentrate (hourly averages)')
plt.xlabel('Correlation')
plt.tight_layout()
plt.show()

Aggregated to hourly, silica concentrate correlates with iron concentrate at -0.80, consistent with the expected inverse relationship between concentrate purity and silica content. Beyond that, the relationships are weak but consistent in direction. Air flow on columns 01 and 03 sits near -0.22, froth levels between -0.12 and -0.19, and ore pulp pH at -0.15. All negative: more of each goes with less impurity, which is what the process is designed to do. One variable breaks the pattern. Amina flow correlates at +0.17, the largest positive value in the set. Amina is the reagent dosed specifically to strip impurities away, so more of it should mean less silica, not more.

Shifting by row position would treat the hour before the recording gap as adjacent to the hour after it. Reindexing onto a complete hourly timeline inserts the missing hours as blanks, which pandas excludes from the correlation. The effect proves negligible here, changing each figure by less than 0.001, but it removes the ambiguity.

hourly_full = hourly.reindex(pd.date_range(hourly.index.min(), hourly.index.max(), freq='h'))

print('Hours on a complete timeline:', len(hourly_full))

Hours on a complete timeline: 4415

print('Does amina now predict silica later?')
for lag in [1, 2, 3, 6, 12]:
    r = hourly_full['Amina Flow'].corr(hourly_full['% Silica Concentrate'].shift(-lag))
    print(f'  amina at t  vs  silica at t+{lag}h: {r: .3f}')

Does amina now predict silica later?
  amina at t  vs  silica at t+1h:  0.145
  amina at t  vs  silica at t+2h:  0.111
  amina at t  vs  silica at t+3h:  0.084
  amina at t  vs  silica at t+6h:  0.052
  amina at t  vs  silica at t+12h:  0.063

print('Does silica earlier predict amina now?')
for lag in [1, 2, 3, 6, 12]:
    r = hourly_full['Amina Flow'].corr(hourly_full['% Silica Concentrate'].shift(lag))
    print(f'  silica at t-{lag}h  vs  amina at t: {r: .3f}')

Does silica earlier predict amina now?
  silica at t-1h  vs  amina at t:  0.194
  silica at t-2h  vs  amina at t:  0.230
  silica at t-3h  vs  amina at t:  0.255
  silica at t-6h  vs  amina at t:  0.190
  silica at t-12h  vs  amina at t:  0.147

Testing the timing of the amina relationship helps explain the unexpected positive correlation. Comparing current amina flow against silica in subsequent hours, the correlation declines from 0.145 at one hour to 0.052 at six hours. Reversing the comparison strengthens it: silica measured one hour earlier correlates with current amina flow at 0.194, two hours earlier at 0.230, and three hours earlier at 0.255, before weakening at six and twelve hours. The relationship is therefore strongest when silica leads amina by approximately three hours. This timing is consistent with reactive dosing, in which higher silica readings are followed by increased amina flow, although delayed laboratory reporting, process residence time, or another shared process factor could produce a similar pattern.

Conclusions

Four things came out of this analysis.

The dataset is not continuous. Thirteen days are missing from March, and no null check reveals it because the hours were never recorded rather than left blank. March is represented by ten calendar days rather than thirty.

The quality measurements are hourly lab results, repeated across the 180 sensor readings that fall within each hour. Correlating them against 20-second data compares real measurements against padding, and produces results that cannot be trusted. Averaging to hourly resolution is a precondition for any analysis of this dataset, not an optional refinement.

Iron concentrate is tightly controlled, varying by only 1.7% around its mean across six months. Silica concentrate varies by 48.3% and offers a more sensitive target for studying how the process performs.

Amina flow correlates positively with silica, which is backwards for a reagent meant to remove impurities. Testing the direction helps explain it: the relationship is strongest looking backwards, peaking when silica is measured three hours before the dosing. The timing is consistent with reactive dosing, though it does not establish causation.

Limitations

These correlations are weak. A value of 0.255 means the relationship explains only a small share of the variation in amina flow, so this is a signal worth investigating rather than a settled conclusion. The analysis also shows only that one measurement tends to follow another in time, which is not the same as showing that one causes the other. A third factor, such as a change in ore quality, could plausibly drive both. Confirming the interpretation would require operator logs showing when dosing decisions were actually made.

What this would need next

The most useful follow-up would be to identify a leading indicator: a sensor reading that shifts before silica does, early enough to act on. If dosing is genuinely reactive with a two to three hour delay, then anything that reliably moves first is worth more to the plant than any of the correlations reported here.