Tag Archives: KPI

COVID-19 Numeracy: Testing

Posted on by

Current situation: 18 March 2020

The United States continues to struggle with COVID-19 testing. Communities do not have sufficient COVID-19 test kits and they are, effectively, rationing tests. Generally, a patient must satisfy several criteria before a COVID-19 test is authorized and performed. Example criteria include cough, fever (high temperature), shortness of breadth, exposure to an infected individual, or travel to a known COVID-19 hot-spot. A patient may be tested first for influenza, receiving a COVID-19 test after influenza is ruled out, i.e., the patient fails the flu test.

Even in the face of shortage, the number of COVID-19 tests performed each day is steadily increasing albeit slowly.

With the current situation in mind, let’s look at few metrics.

Media reports

The most frequently reported statistics are the cumulative number of confirmed COVID-19 cases and the number of deaths due to coronavirus. A few news organizations report the number of new confirmed cases for the preceeding 24 hour period (AKA “per day”).

Most numerate people realize that “If you test, you will find the disease.” If you test more, then the raw number of confirmed cases will also rise.

Very few news organizations report the total number of tests performed per day as well as the number of new confirmed cases per day. Lacking the number of tests performed for each day, one cannot put the number of new confirmed cases in context. Nor can an investigator determine a trend over time, i.e., , is community mitigation (e.g., social distancing, etc.) controlling the spread of COVID-19?

We should insist on receiving three per-day metrics: the number of tests performed that day, the number of new confirmed cases and the number of negative cases. The number of tests performed will tell us if COVID-19 test capacity has improved or not. Then, I recommend computing, tracking and comparing the ratio of new per-day COVID-19 cases (confirmed positives) divided by the number of tests performed that day. The ratio better indicates the growth (or decline) of the disease in the community.

The Washington State Department of Health does report a daily break down. Their page breaks out confirmed cases by county, by age and by sex. The Department of Health also provides testing information. Unfortunately, the number of individuals tested, positive and negative are cumulative. As of March 17, the test figures are:

  • Positive: 1,012
  • Negative: 13,117
  • Individuals tested: 14,129 (Positive+Negative)
  • Percent positive: 1,012 / 14,129 = 7.7%

Please give us nonaggregated results for each day!

Beware sampling bias

One might be inclined to interpret the daily percentage of new confirmed cases as the rate of incidence of the disease in the community (the population at large). Unfortunately, the pre-testing criteria guarantee a biased sample. We are testing patients for COVID-19 only if they have symptoms, have a certain risk factor and/or do not have the flu. The ratio may overstate the incidence of disease or, worse, understate the incidence by missing asymptomatic, infected individuals.

The lack of COVID-19 test capacity leaves public health officials in a terrible bind. One would like to randomly sample (test) the population at large in order to estimate the number of infected people (symptomatic and asymptomatic) in the community. This is akin to taking a political poll. We need to select randomly (say, 1,600 people) from the community and test the selected individuals for COVID-19. We estimate the total number of infected people by multiplying the proportion of infected people as measured by sampling (“the poll”) times the overall population size.

One needs the right kind of test, of course. We want to measure current, active infections, not past infections that have resolved (recovered). This leads us (almost) to the classic SIR model (susceptible, infected, recovered) for epidemics — yet another topic! Also, time to ask the Google about “real-time reverse transcription – polymerase chain reaction (RT-PCR)” and COVID-19 testing. Who said self-isolation was uninteresting? 🙂

From statistics theory, 1,600 tests has a margin of error of plus/minus 2.45%. 1,600 tests is a modest number and, unfortunately, we can barely perform medical diagnostic testing let alone conduct a necessary statistical study at this time. Ideally, we would have sufficient capacity to conduct “tracking polls” to determine the overall trend, i.e., are there more (or less) infected people today than last week? Our officials are truly flying blind at the worst possible time.

Recent studies suggest that asymptomatic people are a significant source of new infections. That’s why public health officials are saying, “Act and behave as though you are infected.” We need tests. We need tools.

Stay apart and say healthy — P.J. Drongowski

COVID-19 Numeracy: NYT curve flattening by country

Posted on by

The New York Times published a collection of graphs comparing new COVID-19 cases over time by country. The article asks the question “Which Country Has Flattened the Curve for the Coronavirus?”

If you are interested in COVID-19 numeracy, I suggest reading the comments section. There are good insights there.

Here is my own response.

Thank you for posting your study. I started blogging about “COVID-19 numeracy” in response to the rather poor way media outlets have portrayed the disease numerically. We are in this fight for the long haul (12 to 18 months) and need metrics that guide our actions.

Please listen to the recommendations in this comment section. Many of us have spent years in measurement and statistics.

Raw numbers, e.g., the number of (new) confirmed cases, are not always meaningful or useful. We know that the number of new confirmed cases per-day will rise dramatically as testing increases. The number of new confirmed cases needs to be “normalized” against the number of tests performed. I suggest tracking the ratio of new confirmed cases divided by the number of tests.

I strongly agree that we need to track the progress of COVID-19 on a daily basis. (A seven day moving average is a good idea.) The total (cumulative) number of cases/deaths — as tracked and reported by most media outlets — will not be useful 2, 3, 4 months into the crisis, especially when there will be “waves” of resurgence and subsidence. We need to understand the dynamics of the pandemic.

Health authorities need to report the number of tests per day, the number of positive cases for that day and the number of negative cases. The raw number of tests per day will tell us if authorities are meeting their commitment to increase the number of tests and allow us to compute ratios, etc.

Should we ever get to the point of spare testing capacity beyond diagnosis, we need to conduct periodic community studies, something akin to a political poll. Take a random sample of the community and determine the number of symptomatic and asymptomatic cases (by age, by sex, etc.) Such polling will allow us to track the actual infection rate in the population at large.

COVID-19: KPI

Posted on by

First off, I would like to say how sad I am for those who have lost their lives to COVID-19, offering my support and empathy to their loved ones. We live in Sonomish County, Washington, not very far from King County and Kirkland. We’ve already seen the devastation which COVID-19 can wreak on care facilities for elderly people. This disease is all too real.

By now, you’ve heard the phrase “flattening the curve” and have probably seen the graph from the CDC (below). Our global goal is to slow the spread of COVID-19 through the population such that severe cases do not overwhelm the health care system. In a nutshell, the health care system has a fixed number of beds, doctors, nurses, caretakers, respirator, ventilators, etc. When capacity is exhausted, the system cannot treat all incoming patients (COVID-19 plus the regular, on-going stream of emergency situations like heart attacks, strokes, etc.) and, quite frankly, people will die.


Source: CDC, Drew Harris (Connie Hanzhang Jin/NPR)

The graph has done a good job of educating us as to the need for social distancing, hygiene, and other measures which slow the spread of disease.

It also suggests a metric — a key performance indicator (KPI) — which can tell us how well we are doing. A KPI is a measurable value that demonstrates how effectively an organization is achieving its objective(s). It’s important to note that a KPI can and should be applied at different levels in the organization. We also need to know how the KPI changes over time.

Let’s consider a KPI which measures the number of all critical care patients above or below the capacity of the health care system (below). This delta tells us if we are successfully suppressing the spread of COVID-19 or not. We must measure all critical care patients because they all are vying for the same medical personnel, beds and equipment. If the KPI is positive, i.e., the number of critical care cases exceeds capacity, then we are failing. If the KPI is negative, then we are succeeding.

COVID-19 Key Performance Indicator (KPI)

A national KPI value is only somewhat useful. The KPI needs to be measured at the state level and regional (metro) level. State-level values are useful only for small states such as Rhode Island with one major population center. Regional-level values are more useful in large states like California or Washington with two or more major population (and health care) centers. The U.S. is a very large country and health care capacity in West Virginia, for example, is not available to residents in San Francisco. Thus, regional numerical break out is required.

We need to track the KPI over time. The trend will tell us if we are successfully supressing the spread of COVID-19 or not. Tracking by region over time will tell us if hot spots are cooling off or if new hot spots are developing.

Why am I proposing this KPI, especially now? As a people, we need to walk a fine line between actionable concern and fearful panic.

I see tables and graphics in the media which tally the total number of confirmed cases and the total number of deaths. Yes, we mourn the loss of our neighbors. If we have even an ounce of humanity, can we not? I agree that such figures convey the sense of urgency needed to motivate new behavior that slows the spread of disease.

However, the total number of confirmed cases and fatalities alone are flawed measures for decision making. With respect to confirmed cases, the number can only go up — drastically — as we ramp up testing. Media need to at least report the number of people tested each day along with the number of negative results as well as the number of new confirmed cases. Even then, perhaps only the ratio of new confirmed cases to the number of new tests is truly meaningful.

Just to be clear, “each day” means the number of people tested, confirmed positive and negative in a specific 24 hour period, not an accumulated tally to date.

Aggregated and accumulated measurements are not practical and may be unnecessarily misleading. News media please take notice.

We also need to break out and track new critical COVID-19 cases which require hospitalization. These are the people who need the health care system. By tracking this number, we will see if the load on the health care system is trending upward toward catastrophe or trending downward successfully. As much as I love my younger and/or healthier brothers and sisters, if they are successfully recovering in self-isolation, then they are not loading the health care system with negative implications, and possibly death, for critical care patients.

I apologize to anyone who may feel offended by my frank discusion. I’m trying to come to grips with all of the information thrown at me by media outlets. Hopefully, you will find this approach to be useful. Even if you do not adopt my proposal, I hope to have started a practical discussion.

Wishing you safe passage — P.J. Drongowski