Six Sigma in Measurement Systems: Evaluating the Hidden Factory

OK

Inputs

Operation

Inspect NOT OK

First Time Correct

Rework
Hidden Factory

Scrap

Time, cost, people

Director, Six Sigma

Bill Rodebaugh
GRACE
slide 1

Objectives

The Hidden Factory Concept

What is a Hidden Factory? What is a Measurement Systems Role in the Hidden Factory?

Review Key Measurement System metrics including %GR&R and P/T ratio Case Study at W. R. GRACE

Measurement Study Set-up and Minitab Analysis Linkage to Process Benefits of an Improved Measurement System

How to Improve Measurement Systems in an Organization

slide 2

The Hidden Factory -- Process/Production

OK

Inputs

Operation
Rework
Hidden Factory

Inspect NOT OK

First Time Correct

Scrap

Time, cost, people What Comprises the Hidden Factory in a Process/Production Area? Reprocessed and Scrap materials -- First time out of spec, not reworkable Over-processed materials -- Run higher than target with higher than needed utilities or reagents Over-analyzed materials -- High Capability, but multiple in-process samples are run, improper SPC leading to over-control

slide 3

The Hidden Factory -- Measurement Systems

OK

Sample Inputs

Lab Work
Re-test
Hidden Factory

Inspect NOT OK

Production

Waste

Time, cost, people

What Comprises the Hidden Factory in a Laboratory Setting?

Incapable Measurement Systems -- purchased, but are unusable due to high repeatability variation and poor discrimination Repetitive Analysis -- Test that runs with repeats to improve known variation or to unsuccessfully deal with overwhelming sampling issues Laboratory Noise Issues -- Lab Tech to Lab Tech Variation, Shift to Shift Variation, Machine to Machine Variation, Lab to Lab Variation

slide 4

The Hidden Factory Linkage

Production Environments generally rely upon inprocess sampling for adjustment As Processes attain Six Sigma performance they begin to rely less on sampling and more upon leveraging the few influential X variables The few influential X variables are determined largely through multi-vari studies and Design of Experimentation (DOE) Good multi-vari and DOE results are based upon acceptable measurement analysis

slide 5

Objectives

The Hidden Factory Concept

What is a Hidden Factory? What is a Measurement Systems Role in the Hidden Factory?

Review Key Measurement System metrics including %GR&R and P/T ratio Case Study at W. R. GRACE

Measurement Study Set-up and Minitab Analysis Linkage to Process Benefits of an Improved Measurement System

How to Improve Measurement Systems in an Organization

slide 6

Possible Sources of Process Variation

Observed Process Variation

Actual Process Variation

Measurement Variation

Long-term Process Variation

Short-term Process Variation

Variation w/i sample

Variation due to gage

Variation due to operators

Repeatability

Calibration

Stability

Linearity

2 Observed Pr ocess 2 Actua l Pr ocess 2 Measuremen t System 2 Measuremen t System 2 Re peatabilit y 2 Re producibility
We will look at repeatability and reproducibility as primary contributors to measurement error
slide 7

How Does Measurement Error Appear?

Actual process variation No measurement error
Frequency
15

LSL

USL

10

0 30 40 50 60 70 80 90 100 110

Process
15

Frequency

Observed process variation With measurement error

LSL
10

USL

0 30 40 50 60 70 80 90 100 110

Observ ed

slide 8

Measurement System Terminology

Discrimination - Smallest detectable increment between two measured values Accuracy related terms True value - Theoretically correct value Bias - Difference between the average value of all measurements of a sample and the true value for that sample Precision related terms Repeatability - Variability inherent in the measurement system under constant conditions Reproducibility - Variability among measurements made under different conditions (e.g. different operators, measuring devices, etc.) Stability - distribution of measurements that remains constant and predictable over time for both the mean and standard deviation Linearity - A measure of any change in accuracy or precision over the range of instrument capability

slide 9

Measurement Capability Index - P/T

Precision to Tolerance Ratio

515 . * MS P/T Tolerance

Usually expressed as percent

Addresses what percent of the tolerance is taken up by measurement error Includes both repeatability and reproducibility

Operator x Unit x Trial experiment

Best case: 10% Acceptable: 30%

Note: 5.15 standard deviations accounts for 99% of Measurement System (MS) variation. The use of 5.15 is an industry standard.

slide 10

Measurement Capability Index - % GR&R

%R & R

Observed

MS
Pr ocess Variation

x 100

Usually expressed as percent

Addresses what percent of the Observed Process Variation is taken up by measurement error %R&R is the best estimate of the effect of measurement systems on the validity of process improvement studies (DOE) Includes both repeatability and reproducibility As a target, look for %R&R < 30%

slide 11

Objectives

The Hidden Factory Concept

What is a Hidden Factory? What is a Measurement Systems Role in the Hidden Factory?

Review Key Measurement System metrics including %GR&R and P/T ratio Case Study at W. R. GRACE

Measurement Study Set-up and Minitab Analysis Linkage to Process Benefits of an Improved Measurement System

How to Improve Measurement Systems in an Organization

slide 12

Case Study Background

Internal Raw Material, A1, is necessary for Final Product production

Expensive Raw Material to produce produced at 4 locations Worldwide Cost savings can be derived directly from improved product quality, CpKs Internal specifications indirectly linked to financial targets for production costs are used to calculate CpKs If CTQ1 of A1 is too low, then more A1 material is added to achieve overall quality higher quality means less quantity is needed this is the project objective

High Impact Six Sigma project was chartered to improve an important quality variable, CTQ1 The measurement of CTQ1 was originally not questioned, but the team decided to study the effectiveness of this measurement

The %GR&R, P/T ratio, and Bias were studied Each of the Worldwide locations were involved in the study

Initial project improvements have somewhat equalized performance across sites. Small level improvements are masked by the measurement effectiveness of CTQ1

slide 13

CTQ1 MSA Study Design (Crossed)

Site 1 Lab

Site 2 Lab

Site 3 Lab

Site 4 Lab

Site 1 Sample 1 Site 1 Sample 2

Site 2 Sample 1..

Op 1 Op 2 Op 3

T1 T2

6 analyses/site/sample 2 samples taken from each site 2*4 Samples should be representative Each site analyzes other sites sample. Each plant does 48 analyses 6*8*4=196 analyses
slide 14

Surface Area

CTQ1 MSA Study Results (Minitab Output)

Components of Variation
120 100 890 %Contribution %Study Var %Tolerance 840 790 740 Gage R&R Repeat Reprod Part-to-Part

Gage name: Date of study: Reported by: Tolerance: Misc:

Z-14 MSA JULY 2002 All Labs 110

Response By Sample

Percent

80 60 40 20 0

Sample
890 840

R Chart by Operator
100
CB1 CB2 CB3 LC1 LC2 LC3 V1 V2 V3 W1 W2 W3

Response By Operator

Sample Range

50

UCL=52.45 790 R=16.05

0 0

LCL=0

740

Oper

CB1 CB2 CB3 LC1 LC2 LC3 V1 V2 V3 W1 W2 W3

Xbar Chart by Operator

900
CB1 CB2 CB3 LC1 LC2 LC3 V1 V2 V3 W1 W2 W3

Operator*Sample Interaction
900 UCL=851.5 Mean=821.3

Operator
CB1 CB2 CB3 LC1 LC2 LC3 V1 V2 V3 W1 W2

Sample Mean

Average

850

850

800

LCL=791.1

800

750 0

Sample

slide 15

CTQ1 MSA Study Results (Minitab Session)

Source Sample DF 7 SS 14221 MS 2031.62 F 5.0079 P 0.00010

Operator
Operator*Sample Repeatability Total

11
77 96 191

53474
31238 26125 125058

4861.27
405.68 272.14

11.9829
1.4907

0.00000
0.03177

%Contribution
Source Total Gage R&R Repeatability VarComp 617.39 272.14 (of VarComp) 90.11 39.72

Reproducibility
Operator Operator*Sample Part-To-Part

345.25
278.47 66.77 67.75

50.39
40.65 9.75 9.89

Sample, Operator, & Interaction are Significant

slide 16

CTQ1 MSA Study Results

Site
All Site 1 Site 2 Site 3 Site 4

%GRR
94.3 (78.6 100)* 38.9 (30.0 47.6) 91.0 (70.7 100) 80.0 (60.8 94.8) 98.0 (64.8 100)

P/T Ratio
116 29 96 79 120

R-bar
16.05 7.22 17.92 20.37 18.67

Mean Equal Variances Differences within Groups (Tukey Comp.)

No (0.004) Yes (0.739) Yes (0.735) Yes (0.158) Yes (0.346) Only 1,2 No Diff. All Pairs No Diff. Only 1,2 Diff. All Pairs No Diff. Only 2,3 No Diff.

*Conf Int not calculated with Minitab, Based upon R&R Std Dev
slide 17

Dotplots of C16 by C17 CTQ1 MSA Study Results (Minitab Output)

(group means are indicated by lines)

Dotplot of All Samples over All Sites

890

840

C16
790 740
LC SA

C17

Site 1

Site 2

Site 3

Site 4

slide 18

WO SA

CB SA

VF SA

CTQ1 MSA Study Results (Minitab Session)

Analysis of Variance for Site Source Site Error Total DF 3 188 191 SS 37514 87518 125032 Individual 95% CIs For Mean MS 12505 466 F 26.86 P 0.000

Based on Pooled StDev

Level Site 1 Site 2 Site 3 Site 4 N 48 48 48 48 Mean 824.57 819.42 800.98 840.13 StDev 15.38 22.11 20.75 26.58 (---*---) (---*---) -+---------+---------+---------+----Pooled StDev = 21.58 795 810 825 840 -+---------+---------+---------+----(---*---) (---*---)

Site and Operator are closely related

slide 19

Per

40 20

790

740 CTQ1 MSA Study Results (Minitab Output) 0 Gage R&R

X-bar R of All Samples for All Sites R Chart by Operator

V1 V2 V3 W1 W2 W3

Repeat

Reprod

Part-to-Part

Sample
890

100

CB1 CB2 CB3 LC1 LC2 LC3

50

UCL=52.45

R=16.05 0 0 LCL=0

Discrimination 840 Index is 0, however can 790 probably see 740 differences of 5
Oper
900

Sample Range

CB1 CB2 C

Xbar Chart by Operator

900
CB1 CB2 CB3 LC1 LC2 LC3 V1 V2 V3 W1 W2 W3

Sample Mean

Mean=821.3 800 LCL=791.1

750 0

Average

850

UCL=851.5

Most of the 850 samples are 800 seen as noise

Sample
1

slide 20

Pe

50

CTQ10 MSA Study Results (Minitab Output)

Gage R&R Repeat Reprod Part-to-Part

X-bar R of All Samples for Site 4 R Chart by WO OP

70 60 50 40 30 20 10 0 0
W1 W2 W3

Sample Range

UCL=60.99

R=18.67 LCL=0

Xbar Chart by WO OP
900
W1 W2 W3

Sample Mean

UCL=875.2 850 Mean=840.1 LCL=805.0 0 Mean differences are seen in X-bar area

800

Most of the samples are seen as noise

slide 21

R=17.92 CTQ1 MSA Study Results Process Linkage 0 LCL=0 760 Site 2 Example 0 LC OP LC1

Sampl

Xbar Chart by LC OP
860 850 840 830 820 810 800 790 780 0
LC1 LC2 LC3

LC OP*Sa
UCL=853.1 850 840

Sample Mean

Mean=819.4

MSA Study 820 Results with 810 800 Mean = 819.4

830 790 1 2 3

LCL=785.7

I and MR Chart for TSA (t)

1 1 1 1 1 2 1

1000

Average
UCL=899.2 Mean=832.5 LCL=765.8 400 UCL=81.95

Sample

Individual Value

11 6 2 2 2 2 6 26 6 6 6 2 2 2 2 2 5 5

900

4 800 6 700 6 2 4 2 2 6 1 0 100

1 300

2002 Historical Process Results with Mean = 832.5

Subgroup

200

150

g Range

1 100

1 1 Selected Samples are Representative 1 1 11 1 11 1 1

slide 22

50

CTQ1 MSA Study Results Process Linkage I and MR Chart for TSA (t) Site 2 Example
810 760 Gage R&R Repeat Reprod Part-to-Part

Perc

1000

Sample

Individual Value

900

100

1 1

R Chart by LC 1 1OP 1
LC1

By LC OP
LC3

1LC2

Sample Range

4 800 50 6 6 2 4 2 2 6 1 700
0

2 2 2 2
UCL=58.54

6 26 6 6 6 2 2 2 2 2

5 5860

UCL=899.2

810

R=17.92 LCL=0

MSA Study Results with Range = 17.92, LCL=765.8 Calc for Subgroup
Mean=832.5 400
LC1 LC2 LC3

Subgroup
0

100

200

300

760

LC OP

150
860 850 100 840 830 820 50 810 800 790 0 780 0

Xbar Chart by LC 1 OP
1 1
LC1

Moving Range Sample Mean

1 11 1

LC OP*Sample Interaction
LC3

LC2

11 1

UCL=853.1

850 840 830 820 810 800 790

1
Average
Mean=819.4

UCL=81.95

2 2

2 2 2 2 2

R=25.08 LCL=0
1 2 3

2 2

2 2 LCL=785.7

Sample

2002 Historical LC O L Process L L Results with Range = 25.08 Calc for pt to8 pt 4 5 6 7

When comparing the MSA with process operation, a large percentage of pt-to-pt variation is MS error (70%) --- a back check of proper test sample selection
slide 23

CTQ1 MSA Study Results Process Linkage Site 2 Example

Key issue for Process Improvement Efforts is When will we see change?

Initial Improvements to A1 process were made Control Plan Improvements to A1 process were initiated Site 2 Baseline Values were higher than other sites Small step changes in mean and reduction in variation will achieve goal

How can Site 2 see small, real change with a Measurement System with 70+% GR&R? Use Power and Sample Size Calculator with and without impact of MS variation. Lack of clarity in process improvement work, results in missed opportunity for improvement and continued use of non-optimal parameters
slide 24

CTQ1 MSA Study Results Process Linkage Site 2 Example

2-Sample t Test Alpha = 0.05 Sigma = 22.23 Target Power 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 Actual Power 0.9000 0.9002 0.9002 0.9001 0.9020 0.9023 0.9007 0.9018 0.9017 0.9016 Difference 2 4 6 8 10 12 2-Sample t Test Alpha = 0.05 Sigma = 6.67 Target Power 0.9000 0.9000 0.9000 0.9000 0.9000 0.9000 Actual Power 0.9011 0.9036 0.9015 0.9074 0.9188 0.9361 Sample Difference 2 4 6 8 10 12 14 16 18 20 Size 2117 530 236 133 86 60 44 34 27 22 Sample Size 192 49 22 13 9 7

14
16 18 20

5
4 4 3

0.9000
0.9000 0.9000 0.9000

0.9156
0.9091 0.9555 0.9095

Simulated Reduction of Pt to Pt variation by 70% decreases time to observe savings by over 9X.
slide 25

CTQ1 MSA Study Results Process Linkage Site 2 Example Benefits of An Improved MS

Realized Savings for a Process Improvement Effort

For A1, an increase of 1 number of CTQ1 is approximately $1 per ton Change of 10 numbers, 1000 Tons produced in 1 month (832 842) $1 * 10 * 1000 = $10,000

More trust in all laboratory numbers for CTQ1 Ability to make process changes earlier with R-bar at 6.67

Previously, it would be pointless to make any process changes within the 22 point range. Would you really see the change?

As the Six Sigma team pushes the CTQ1 value higher, DOEs and other tools will have greater benefit

slide 26

Objectives

The Hidden Factory Concept

What is a Hidden Factory? What is a Measurement Systems Role in the Hidden Factory?

Review Key Measurement System metrics including %GR&R and P/T ratio Case Study at W. R. GRACE

Measurement Study Set-up and Minitab Analysis Linkage to Process Benefits of an Improved Measurement System

How to Improve Measurement Systems in an Organization

slide 27

Measurement Improvement in the Organization

Initial efforts for MS improvement are driven on a BB/GB project basis

Six Sigma Black Belts and Green Belts Perform MSAs during Project Work Lab Managers and Technicians are Part of Six Sigma Teams Measurement Systems are Improved as Six Sigma Projects are Completed

Intermediate efforts have general Operations training for lab personnel, mostly laboratory management

Lab efficiency and machine set-up projects are started The %GR&R concept has not reached the technician level

Current efforts enhance technician level knowledge and dramatically increase the number of MS projects

MS Task Force initiated (3 BBs lead effort) Develop Six Sigma Analytical GB training All MS projects are chartered and reviewed; All students have a project Division-wide database of all MS results is implemented

slide 28

Measurement Improvement in the Organization

Develop common methodology for Analytical GB training

Six Sigma Step Define Action Target measurement system for study Identify KPOVs Identify KPIVs Evaluate KPOV performance Measurement System Analysis Reduce Reproducibility Reduce Repeatability Reduce Operator or Instrument Bias Final Report Control Plan for KPIVs Typical Six Sigma Tools Used Project Charter

Measure

Analyze Improve

Control

Soft tools: Process Map, Cause & Effect Matrix, FMEA Stat tools: Minitab Graphics, SPC, Capability Analysis Gage R&R, ANOVA, Variance Components, Regression, Graphical Interpretation Soft tools: Fishbone Diagram, Focused FMEA Stat tools: D-Study, t-Tests and Regression, Design of Experiments SPC, Reaction Plans, Control Plans, ISO synergy, Mistake Proofing

slide 29

Final Thoughts

The Hidden Factory is explored throughout all Six Sigma programs One area of the Hidden Factory in Production Environments is Measurement Systems Simply utilizing Operations Black Belts and Green Belts to improve Measurement Systems on a project by project basis is not the long term answer The GRACE Six Sigma organization is driving Measurement System Improvement through:

Tailored training to Analytical Resources Similar Six Sigma review and project protocol Communication to the entire organization regarding Measurement System performance As in the case study, attaching business/cost implications to poorly performing measurement systems

slide 30