Last month, I attended the International Energy Program Evaluation Conference (IEPEC) in San Diego after two long reschedules due to COVID-19. Attending the conference was refreshing on both a professional and personal level. The evaluation community is tight-knit with a collegial attitude. And it should be—it’s a small world where the players in the game often switch uniforms. Your competitor one year may be your colleague (or client) the next. IEPEC’s magic is that it becomes an opportunity for players in the game to set aside whatever jerseys they may wear and focus on moving the industry forward by sharing knowledge and making connections.
Equity, distributed energy resources, and non-energy benefits were among the themes for the sessions. From my perspective, the one thing missing the most was the focus on the customer in demand-side management (DSM) program evaluations. This omission made me reflect on my career, everything I learned in evaluations, and what I’m helping utilities achieve now with a data-driven approach to audience segmentation.
What do DSM program evaluations reflect about how utilities value customers?
Evaluators do a great job showing how the program is tracking against goals for the utility and regulators. They don’t, however, have any focus on evaluated savings or expected savings at the individual level. This paints a picture of customer participants simply as a means to an end. If anything, participants may get a high-level email that shares how many trees their participation was worth, which feels like a pat on the head. What about an acknowledgment of how experiences shared between customers, their neighbors, and the utility drive efficiency gains?
Everything I saw presented at IEPEC seemed to stop at the program level (or at a regional level) rather than bringing impacts all the way down to individual customers. There’s an assumption that evaluators can do their thing and someone else can package up the most relevant information out of aggregate estimates to relay back to the participating customers and use in marketing campaigns for recruiting new participants. What does this look like? Let’s consider a hypothetical:
Thank you for partnering with Super Awesome Utility to weatherize your home! We estimate that you saved $35 on your last bill. With colder weather coming, you should save $40 on your next bill, thanks to the reduced heat loss from your home. Based on your usage, we expect you could save an additional $15 on your winter bills and $10 on your summer bills by enrolling in Program XYZ.
You have to wonder, Are the estimates in this example specific to the individual customer or are they averages that may be way off, leaving the customer feeling like they were misled by Super Awesome Utility? Is Program XYZ really the optimal next program for this customer? Is the message simply words and numbers, or does it have compelling and intuitive visualizations? After answering these questions, I think you’d agree that the example above misses the mark.
Why stop at evaluating program impacts?
I started my career at the US Energy Information Administration, applying survey research and statistical modeling to provide stakeholders in the energy industry with characteristics of energy users from different sectors of the US economy and how much energy they use to serve their needs. This involved energy efficiency at a macro level, but it wasn’t until I switched jobs that I learned more about energy efficiency and, more generally, DSM as a utility program. The whole notion of utility DSM programs was extremely confusing. Why would a utility go through all the trouble of having programs that seemed to reduce revenue?
I learned over the years more about how this all worked and why DSM programs are so valuable. They include a whole ecosystem of entities—electric and gas utilities, state commissions, wholesale energy markets, intervenors, program implementers, trade allies, and, yes, evaluators. Evaluation—impact evaluation, to be specific—ended up being my area of focus, because that’s where the data and statistical modeling tend to live. And the challenge of teasing out the measurable effects of these programs was endlessly fascinating. After many years in the evaluator role, I learned the state of the art from colleagues, from attending conferences like IEPEC, and from reading the presenter papers. But program implementation was mostly foreign to me—behind the big curtain.
Evaluators excel at applying appropriate methodologies for estimating program impacts, useful from the 10,000-foot view. Why not also go down to ground level, providing estimates specific to individual customers who participated in, or are considering participating in, the programs? Doing so would help make sure that program implementers have the best information at their fingertips and can focus on using it effectively to optimize the programs offered.
Rethinking your DSM strategy to reach an Audience of One
In my current role at E Source, we see bridging program impacts and target marketing as a critical step if utility programs want to complete the feedback loop from evaluation to implementation, pushing toward impactful, cost-effective programs that also drive customer satisfaction.
Our DSM projects tend to be on top of the implementation-evaluation firewall, rather than on either side of it. Here are some examples of how our data science solutions help utilities focus on customers as individuals:
- Measure demand response impacts rapidly (within minutes of getting usage data) by customer to enable feedback when it matters most
- Segment nonparticipating customers for programs according to their expected impact contribution if they were to enroll in a particular program
- Find the right customer for the right program (that is, which customers want to be part of the program and are expected to be high-impact contributors?
- Prioritize program marketing to keep customers from being overwhelmed with choices
These are just a few of the solutions in the E Source Audience of One portfolio. Think of us as the data science engine to power your DSM strategy and operations and the copilot to help you achieve the success you’re driving toward.