The Region 4 Stork (R4S) Collaborative Project
Part 2: First Generation Productivity Tools
Click CC box for captions; full transcript is below.
Published: June 2013Print Record of Viewing
Accurate identification of newborns with metabolic disease can significantly improve patient outcomes. Conversely, a missed diagnosis can result in significant morbidity and may even result in death. While a false-positive diagnosis does not carry the burden of increased morbidity or mortality, there are social and psychological costs that may generate significant harm. The Region 4 Stork Collaborative was developed to improve detection of true positive cases of metabolic disease and improve accurate diagnosis. The R4S project uses Mayo-developed software that provides postanalytical interpretation of complex metabolic profiles. The R4S project offers physicians worldwide the opportunity to utilize this software to analyze their patients’ test results, and compare them with other locations’ results.
R4S Collaborative Project Part 2 demonstrates the use of first generation productivity tools.
Presenter: Piero Rinaldo, MD, PhD
- Co-director of the Biochemical Genetics Laboratory
- Professor of Laboratory Medicine and Pathology
- T. Denny Sanford Professor of Pediatrics at Mayo Clinic
Questions and Feedback
TranscriptDownload the PDF
Thank you for the introduction. This presentation is the second segment of a 6-part series describing the products and clinical tools of a newborn screening quality improvement project called Region 4 Stork, or R4S.
I have a disclosure to make: a provisional patent application related to some of the content of this series has been submitted by Mayo Clinic. The title of the application is “Computer-Based Dynamic Data Analysis.”
To briefly recap the first presentation, 2 types of tools were introduced: the plot by condition, shown here on the left side, where all markers for 1 condition can be evaluated to establish clinical significance, and the plot by disease range, where a single analyte for a single condition is presented as a separate disease range for each individual site that has contributed at least 1 case with the chosen condition.
The summary table that is shown in this slide compares the content and purpose of the 2 tools already discussed with 5 more types of first generation tools:
They are the plot by marker, the plot by target range, the score card report, the interactive scatter plot and finally the analyte comparison tool. The objective of this presentation is to illustrate how the tools could be deployed to describe a laboratory own status and performance in a context of objective and evidence-based peer comparison, 1 that is always up-to-date and available on demand.
The first tool is the plot by marker. This tool could be described as the opposite combination of the plot by condition, because it shows the disease ranges of just 1 marker, but in all conditions regardless of the achievement of clinical significance. Just like all other tools, access to the plot by marker is provided by a link available on the web home page.
In the selection window, we continue to use as a model C14:1, the primary marker of VLCAD deficiency. The analyte type is, of course, acylcarnitines.
C14:1 is selected from the analyte drop down menu as shown in this animation. As a final step, 1 or all condition types can be selected. It is recommended to select at first all 3 types of conditions (amino acid, fatty acid, and organic acid disorders) to cover the full spectrum of conditions.
After clicking “show chart”, the following image is displayed. Most elements are kept consistent from tool to tool, like the representation of the cumulative reference range of C14:1, shown both as a green shade as well as a box plot of the 5 percentile values that were introduced in the previous presentation.
Disease ranges are color coded, bright red represents conditions where C14:1 is informative according to the rule that the median value, shown as a white circle, is positi1d above the upper limit of the reference range, shown as a green shade. The insert shows an expanded view of the C14:1 informative disease ranges: VLCAD deficiency and VLCAD carriers, glutaric academia type II (abbreviated as GA-2), and 2 conditions which are shown together because their biochemical phenotypes are essentially identical. They are long-chain 3-hydroxy acyl-CoA dehydrogenase deficiency, LCHAD, and trifunctional protein deficiency, or TFP.
All other conditions are shown as gray boxes, the respective medians are well within the reference limits.
What is a clinical utility of the plot by marker tool? The plot by marker is very useful to recognize the extent of differential diagnosis that is required to properly interpret an informative result for a given analyte. It is important to emphasize here that testing by mass spectrometry for the recommended newborn screening uniform panel involves a large number of markers and calculated ratios, more than 100 overall. The vast majority of them require a differential diagnosis that could span from just 2 to 13 conditions, with an average of 5. In other words, when a result is abnormally high or low, a direct relationship to just 1 possible diagnosis should be considered the exception rather than the rule. In our experience, proper consideration of all possible outcomes is indeed essential to properly select cutoff values.
Selection of a cutoff value is best achieved by using the next productivity tool, the plot by target range. This tool is different from the previous 1 in 2 aspects: it shows only clinically significant disease ranges and also adds between reference and disease ranges a new element, the cutoff disease range. The link to access the plot by target range, like all other tools, is found in the website homepage.
A different analyte has been selected as the model to demonstrate the purpose of this tool. Xle is an abbreviation commonly used to indicate the mixture of 2 branch chain amino acids that have the same molecular weight (isoleucine and leucine). In a testing mode that does not involve front-end chromatographic separation, isobaric compounds like these 2 and also other minor components are measured together. Elevated concentrations of these 2 markers combined are informative for just 1 condition, maple syrup urine disease, shown in the slide as MSUD. From the left side, the 3 sections of this plot correspond to the reference, cutoff, and disease range of isoleucine/leucine, respectively. The cumulative percentiles are based on the data contributed by 115 laboratories worldwide, equivalent to several millions of newborns; the cutoff range is calculated from 127 independent values; the total number of true positive cases is 283. These cases have been uploaded by 87 different programs, each contributing between just 1 to 27 cases, with a median of 2 per site. The ordinate scale (the Y-axis) of all R4S plots can be adjusted as needed. For example, changing the scale from the default level of greater than 3,000 nmol/mL to approximately 800 is done by a click of the mouse at the desired level.
Once the scale has been reduced, it is possible to better appreciate these ranges, but also to introduce the concept of cutoff target range. For a disease where clinical significance is found above the reference population, like in this case, the lower limit corresponds to the 99th percentile of the reference range. The upper limit is equal to the 5th percentile of the disease range. This choice, rather than either the lowest value observed for a given condition or the 1st percentile of the disease range, is necessary because virtually in every condition there are highly unusual cases with normal or even lower than normal results. These cases would be missed by any cutoff value for isoleucine/leucine a lab would choose, and any attempt to identify then would trigger unmanageable numbers of false- positive cases. The interval between the 2 levels is what is called the CUTOFF TARGET RANGE. As I mentioned earlier, there is 1 important and interesting observation to be made here: the MEDIAN value of the 127 cutoffs used in screening practice worldwide is ABOVE the target range. This situation is likely to cause multiple false-negative events that could be avoided by just selecting a cutoff values that lies within the target range. There is another informative marker of MSUD, the branched chain amino acid valine. Could valine compensate for the poor specificity of isoleucine/leucine?
Indeed that is not the case. The same plot for valine shows an even greater overlap between reference and disease ranges, and also a target range that had to be adjusted to compensate for this situation. The median of the cutoff range is even higher than in the case of isoleucine/leucine revealing a trend of selecting cutoff values on the sole basis of the reference range, therefore penetrating very deep in the MSUD disease range with a degree of overlap of almost 70%.
The concern for the selection of cutoff values for these markers is unfortunately shared by many others. Another compelling example is propionylcarnitine, an acylcarnitine species commonly abbreviated as C3. Like in the case of MSUD, we begin with the plot by disease range for a single condition, propionic academia. Just like before, the median of the cutoff range is higher than the target range. This situation is actually more concerning once the question is asked if C3 is informative for other conditions beside propionic acidemia.
It is a fact that high concentrations of C3 could be found in as many as 8 other conditions, shown here only by their abbreviated names. More importantly, the corresponding disease ranges are lower than the one found in propionic acidemia. Clearly, the degree of overlap between all disease ranges and the cutoff range becomes so pervasive that it is not rare to see cutoff values set at a level higher than the median of some disease ranges. Basically, this means that there is a greater than 50% chance that an affected case could be missed by newborn screening, causing false-negative events that would have been possible to avoid by a more in-depth clinical validation of cutoff values.
It should be mentioned that C3 is also informative BELOW the reference range. This plot shows that there are at least 5 conditions where a low concentration of C3 is actually very informative. While 120 laboratories have implemented a high cutoff value for C3, to date only 50 of them have also adopted a second cutoff at the low level. In conclusion, C3 in indeed the marker where the full extent of differential diagnosis should include 13 different conditions.
For a summary of the clinical utility of the plot by target range, consider reading this 2011 publication entitled “Clinical validation of cutoff target ranges in newborn screening of metabolic disorders by tandem mass spectrometry: a worldwide collaborative project”. In this paper, the disease ranges were based on less than 11,000 true-positive cases, equivalent to the status of the project at the end of 2010. While the size of this population study is unprecedented, it should already be considered out of date already. Just 2 years later, at the end of 2012, the count had grown by a third to more than 14,000. Since 2009, an average of 5 new cases are uploaded to the database every single day.
This constant growth is exactly the reason why the next productivity tool has been created. The score card provides a dynamic, instead of a static, report of the cutoff target ranges, one that is kept constantly up to date. The score card summarizes the data of all significant analytes and ratios for all conditions. Access is from a link on the homepage, like all other tools.
This slide shows the selection window of the score card. Please note the additional link shown at the top left corner of the blue panel. By clicking “How to use this tool”, a user can access a detailed procedure that explains the content of this tool and how to use it properly.
Access to this training material is provided in 3 ways: from the menu bar under the heading Documentation, from the home page, and as shown already from the selection window of every tool.
The list of available training documents is shown in this slide. There is a document available for each of the tools presented so far: the plot by condition, the plot by marker, the plot by target range, and of course, the score card.
Going back to the score selection window, the type of analyte is the only choice to be made. Once 1 has been chosen, like in this example, the score card for that type of marker is displayed.
The score card summarizes in a tabular rather than a graphic way all the data shown as box plots in the previous tools. Each section includes a current count of contributions, either sites or cases, and cumulative percentile values: reference ranges by analyte on the left side, the high and low disease ranges on the right side. This section is sorted by individual conditions and also includes 3 additional percentile values: the 5th, the 25th, and the 75th percentile. In the central portion of the score card, there is a summary of high and low cutoff ranges, and finally the high and low cutoff target ranges. If needed, a detailed explanation of the color coding is included in the training material available on the website. Briefly, when the background color of the target range is either yellow or purple, it indicates that there is a complete separation between the reference range and the disease range of the chosen target condition. When the background color is black, it means a manual adjustment was necessary because the 2 ranges do overlap. One of the analytes discussed previously, C3, is shown in this slide. As a reminder, all these ranges are updated instantly after any new submission to the website.
This is another portion of the acylcarnitine score card tool that shows the data for the other analyte we have discussed previously, C14:1. The disease ranges on the right side are condition specific, showing, for example, the differences between VLCAD deficiency and VLCAD carriers.
If amino acids were chosen as analyte type, the following page would appear, showing at the top the branched-chain amino acids that have been used as model to illustrate the plot by target range tool.
To appreciate the magnitude of the information available in this tool, this slide shows the full sets of ranges available for the 4 analyte types: amino acids, amino acid ratios, acylcarnitines and, on the far right, acylcarnitine ratios. As a reminder, the score cards are always up-to-date and, of course, are available on demand.
The next tool presented in this series is called the scatter plot. This plot is very interactive in nature, as it provides a powerful way to investigate the distribution of true-positive results not as cumulative ranges but as individual cases. The plot shows the actual results of any combination of 2 markers, and as many conditions as desired.
This is the entry page of the scatter plot. The model for the demonstration of how to use it is provided again by the 2 branched-chain amino acids, isoleucine/leucine and valine.
A user needs to select the analyte type, the specific analyte for the X and Y axes, the condition type, and finally the condition of interest. To select more than 1 condition, hold the CTRL key while highlighting with the mouse all the conditions to be displayed. For this demonstration, the median value of the 2 cutoff ranges is also included, 278 for isoleucine/leucine and 250 for valine. The scatter plot offers the option to display or hide all other true-positive cases. In this example, it has been deactivated by unclicking the box at the bottom of the window.
When “show chart” is selected, the following image is displayed. The pair of results for each case with MSUD is shown as a blue dot. The scale is automatically set to 20% more of the highest value on record, so the reference limits are compressed at the bottom left corner of the plot. This is easily changed by clicking the mouse once within the plot to reformat the 2 axes. To continue this presentation, both axes are reset to a value of approximately 1,000.
The vertical and horizontal lines of the green area corresponds to the 1% and 99% percentiles of the reference population, the dotted red lines represent the cutoff values entered in the selection page. As a reminder, they are shown on the left side of the slide together with the percentile of the cutoff range they represent, in this case the 50th, which is the median, and the count of values included in the range (128 and 115, respectively). The red lines divide the plot in 4 quadrants. The top right quadrant shows all cases with results exceeding both cutoff values. At the opposite end of the spectrum, the bottom left quadrant shows how many cases would have been missed using the same parameters. Indeed, some of them are documented false-negative events added to the R4S database after clinical ascertainment. The other 2 quadrants show what we may call the mixed results, 1 positive and 1 negative. Clearly, many cases with MSUD had a valine result below the median cutoff value, information that could help in the interpretation of challenging cases.
What happens when different cutoff values are selected? If, for example, we plot the respective 1st percentiles of the cutoff ranges (176 for isoleucine/leucine and 143 for valine) the detection of true positive cases becomes almost perfect, but the overlap with the reference population, the green area, would cause an unacceptably high number of false- positive cases.
At the opposite end of the cutoff ranges, the cutoff values corresponding to the 99% percentiles, which are 492 for isoleucine/leucine and 450 for valine, a majority of affected cases would be missed. This is far from being a hypothetical situation because these are the actual values used by 1 laboratory in its routine screening practice.
In summary, the scatter plot could be viewed as a testing environment where a laboratory could verify its choices of cutoff values against the cumulative experience worldwide. As an example, we consider a laboratory with cutoff values for isoleucine/leucine and valine of 315 and 300, respectively.
The scatter plot clearly shows that at those levels the risk of false-negative events is fairly high considering the high number of cases with 1 or both results below the chosen cutoffs.
The scatter plot is particularly useful in the prospective evaluation of challenging cases. For example, the same laboratory has to provide an interpretation for a newborn with the following results: isoleucine/leucine 300 nmol/mL and valine 283. These values can be entered in the selection page to generate the next image.
The red diamond and the arrow show where this case is located. Although the results are below the cutoff values, there are several true-positive cases with similar and even lower results. This could be a very challenging situation. Fortunately, a review of the demographic information of this patient reveals that at the time of sample collection the newborn was on total parenteral nutrition, or TPN.
When TPN is added to MSUD as the conditions to be displayed, the image is modified as shown here. The orange dots indicate TPN cases, and indeed, this case is located very close to the majority of them. However, the overlap between MSUD and TPN cases remains unresolved.
When the type of alimentation is TPN, the concentrations of additional amino acids could be elevated, particularly phenylalanine, shown as Phe and methionine, shown as Met. The values in this case, like the branched-chain amino acids, can be described as borderline in reference to the median cutoff values: 150 for phenylalanine and 60 for methionine.
However, plotting the phenylalanine and methionine values in MSUD and TPN cases shows an improved separation between the cluster of MSUD cases and the case under evaluation, shown again and a red diamond and by the blue arrow.
As a final step, the introduction of ratios can provide the conclusive evidence to resolve this case. Shown here are 2 ratios, isoleucine-leucine/phenylalanine and methionine/ isoleucine-leucine, the median cutoff values of the R4S project, and the calculated values in this patient, 2.07 and 0.23, respectively.
Considering the position of this case and of the cluster of TPN cases, the separation between MSUD and TPN cases is virtually complete, and this case could be closed with confidence as a nutritional artifact that does not require a referral to follow-up and the collection of a repeat specimen and/or additional laboratory investigations.
The final plot of this presentation is the analyte comparison tool. It allows the single simultaneous evaluation of the basic elements of any analyte in comparison to the collective experience of all other laboratories.
Once again, the model analyte is isoleucine-leucine. The tool shows the range of the cumulative reference percentiles, the anonymized reference percentiles of each individual contributing site, the disease range of all conditions where the chosen analyte is informative, in this case just 1, MSUD, the distribution of individual cutoff values shown next to the disease range to easily appreciate the degree of overlap, the cutoff target range proposed by the R4S project, and the option to highlight the reference percentiles and cutoff value of your own laboratory. The size of the blue or purple diamond is proportional to the number of laboratories that have chosen the same cutoff value. Like all other tools, the analyte comparison tool is kept instantly up to date after any new data submission and is available on demand.
This is the conclusion of the second part of this series. In part 3 we will begin to illustrate how the evidence-based approach and concepts of the first generation tools have been utilized to create the second generation of tools, which have been called the postanalytical interpretive tools of the R4S project.
Thank you for your attention. Please do not hesitate to contact us if you have any questions or requests related to the content of this presentation.