使用RapidMiner进行话量预测

00:03[音乐]来自FirstEnergy的Mike Stansky。我们是公用事业单位。所以，不像很多演讲[笑声]。我看过每个人的，他们非常有市场，非常华丽。而我们的做法是，把尽可能多的信息放进幻灯片中，然后把它们全部倾倒给你们。所以它不那么漂亮，多了很多字，但没关系。我会把这些都过一遍。题目是《话务量预测》。这是它的核心。这就是Clarkston帮助我们制作的时间序列模型。 And Clarkston has these little things over here if you want to see what they did for us. It was actually fantastic. So I will talk about that, the forecast itself, but a lot of this is more on our engagement with Clarkston in our first experience with RapidMiner or rollout of RapidMiner. So sort of pro forma, FirstEnergy is a fully electric utility spanning those states. We have six million customers. We’ve actually been in the business for 112 years, not as FirstEnergy, FirstEnergy only since 1998. So 22 years or so, as FirstEnergy Corp. But that was after the merger of the Ohio and the PA utilities. And then in 2011, we picked up the West Virginia and Maryland utility. And at one time we had a bunch of generation coal-fired power plants, nuclear power plants. We’ve actually fully spun off that business unit as of last year. Actually, they’re fully emerging just this year. It keeps on getting delayed.

01:49我们为全美六百万客户提供服务，我们有——什么?我甚至不记得这个数字了——26.9万英里的电线，所以我们真的是一家运输公司。我们为客户提供电力，并把电力送到他们的家里。满足我们的使命，有远见的电力公司，新员工让他们的生活更光明，环境更好等等。好的。所以我们在2018年收购了Raffan Miner。在那之前，可能很多人都经历过，整个组织的分析人员都在做不同的事情，一群聪明的人。本来要做演讲的是Brian Rearden，他开始组织这个小组。他得到了一个叫量化集团的领导的赞助。我们每个月见一次面，在周五的午休时间讨论数学。 And out of that, we realized we needed a single tool for analytics. We put on our feet, engaged with RapidMiner. And RapidMiner was rolled out in 2018 to about 10 to 20 users in our first round. But most of the work initially was ad hoc and we decided, okay, we need to show that this is going to work, and if we do it ourselves, it’ll probably fail. So let’s engage somebody. We called RapidMiner, RapidMiner sent us to Clarkston, and we engage them to help us with what we selected which was called Volume Forecasting out of our call centers.

03:30同样在2018年，我们开始了RapidMiner用户组。所以我们每个月见一次面，不仅仅是我们20个人。现在已经接近50个了，但说实话，只有25个电话能接通。我们提供一些提示和技巧，询问人们在做什么，当然也会讨论扩展和服务器上的东西，所有维护的东西。然后在19年，我们也开始了黑客马拉松，不是针对RapidMiner的;你可以自带工具。但RapidMiner最终成为了黑客松的一个重要组成部分，因为我们在RapidMiner上快速创建了许多开发原型，并与各种各样的人坐下来，从高管到商业领袖，实时创建他们的愿景。所以这真的很吸引人。我们举办了三个。他们都很有成效。 And really just this year we reached 50 users. Still, most are just doing things outside of production. Outside the server. It’s ad hoc work. ETL, scope out a model, that sort of prototyping. But we do have to fully enterprise production processes, only, two. But what that means is a little tricky at FirstEnergy. But we have a bunch in staging and development, and I’ll talk about that later. And it’s currently used across the organization from our Smart Meter Ops to our Customer Service Contact Center. I’m actually in long-term planning and analytics, but we’re kind of going through some transitioning. I will probably end up in our innovation organization. So we are standing up a separate organization for analytics. So we really don’t have many data scientists; it’s like three of us [laughter], so. But ideally, we’ll have a lot more coming this year.

05:21在呼叫中心，我们有大约500名全职员工，我想说，大部分都是朝九晚五的人，但他们是安排好的员工。然后我们真正地上下工作，召集了一个大约200名承包商的团队来处理这个班次。这是基于我们项目电话将如何进入的日常轮班。这就是我们能够平衡呼叫中心负载的地方。我们感觉有1600万——我甚至没有意识到有这么多——并不是所有的人都直接给了经纪人。一些在我们的IVR中处理，甚至是在IVR中处理的。他们被分散到不同类型的代表中，所以我们的csr要接各种各样的电话。实际上，大多数电话都是计费的。我们几乎所有的电话都在计费，“为什么我的账单这么高?”我付不起账单了。” But we also, certainly during emergencies, power outages, we get high volumes, and that’s when we do things like call in our 200 contractors. So our prior Call Volume Forecasting, we had two. One was sort of our daily hey. It’s just kind of what it looked like last year before. And our monthly, which a bunch of people got in the room and figured out what we need. So really the monthly for two years is for long-term staff. We hire those people for our baseload. And then our daily is to shift with that work. And ultimately, it’s for all these things, scheduling holidays, scheduling short-term labor issues, certainly managing our full-time requirements, and using the call volume forecasts to project are real variable of interest which is our first call resolution.

07:27还有各种各样的kpi。我认为第一个电话解决方案可能是呼叫中心最差的绩效KPI，但它是我们选择的，因为它真的没有谈到满意度。你可以在接到5个电话时高兴地打电话，而一个电话就解决了你的问题时，你就痛苦地打电话。它确实没有达到满足感，但我们用它来衡量满足感。但最终，我们以前的预测是非常人工密集的，而且是在空中完成的。我觉得会是这样的。事实上，我偷走了你的视线。收藏在这里[笑声]。这显示了旧的过程启发式。所以接触的第一部分将Clarkston然后做Alisse说——如果你在她的演讲——早些时候,进来和我们一起坐下来,或者坐下来与业务专家和经历的类型变量,他们认为,他们认为我们做什么预测,当你得到它,你将如何使用它,最重要的事情你的预测。 So ultimately, we looked at just straight-up correlations for abandoned calls, move-in, move-out, implausible bills, all those. And when we’re looking at our daily calling forecast or even our monthly, the correlations are actually really low. The things that really matter are these down here. What was your calls yesterday? What was your calls last week? What was your calls, this type of day last year? And then all the date variables. So the business unit indicated very much accuracy is most important. They want to know what the volume will be.

2008年9月17日所以重点当然是让我们建立一种我说的时间序列，但这不是时间序列模型。它最终成为一个梯度增强的树木模型，这是一个趋势。这是模式匹配的，至少对于每日的预测是这样。因此，短期的，90天的预测我们最终会使用一个机器学习模型，带有时间严重的成分。我们学到了一些技巧。我知道RapidMiner在这方面做了一些改进。但与SARIMA类型的模型相反我相信你们已经研究过SARIMA。是的,好吧(笑)。然后长期使用Holt-Winters的模型，这实际上只是一个模式匹配型模型[听不清]，是的，统计模型。所以如何处理时间序列是棘手的。 So in the past, we were guessing [laughter], so. And after guessing, well, a combination of guessing and what was last year. And that’s what a lot of times year forecast ends up being within a business because that’s what you know, and that’s easy for a manager and leader to get a feel for. And it gives you confidence. You’re like, “Oh, well, can’t be that wrong. I mean it happened last year.” The other option is to follow statistical method, which is a remodel, or Holt-Winters where you are following the trend. Or you can even do a straight regression with something like weather where you actually have some actuals that you’re going to move with all following that your model. Or you can ignore all those and do some sort of machine learning method where you can have the attributes for that specific day in the past and you predict what that next day will be. But normally that involves some sort of windowing of your data and projecting what T zero is based on T minus one, T minus two to my story.

11:14但是当你到达T +24时，你需要t23, t23意味着t22。所以有一些技巧他们做了一项了不起的工作为我们阐明了这些技巧。所以我直接跳到表演上。表现比我们想象的要好得多。我们改进了我们的MAPE。我们用了MAPE，我讨厌MAPE。MAPE是我存在的祸害，因为即使你一天少了一分钱或取消一次，那也应该是零，你的MAPE是无限的。这对我们来说不是很好。当你得到更小的实际值时，MAPE会爆炸。所以绝对均值可能会好一点。 But this is our MAPE where we cut our performance by or our error rates by about half. So it was a significant improvement in accuracy. And the goal is ultimately accuracy for this one. The stated goal was accuracy, and I’ll talk about that later. And the Holt-Winters, which actually this was Alisse who talked before, performs very well. We have some out-of-sample stuff, and it has held up really, really well. This also raised some discussion on the downward trend of our call volume, which is actually a measure of our improvement in our IVR system, dispensing calls and getting them out of individual people’s hands and handling them before they get to a CSR. So our previous method – sorry, checking my time – it was that manual effort and had to be repeated often and someone had to get involved in it and essentially save an Excel file and throw it into ClickView.

我们的新方法实际上是每周运行一次的。我们对短期模式进行再培训和重新部署结果在90天内发布，然后我们每季度进行一次。这实际上是没有安排的;有人按下了季刊的按钮。所有这些都会在ClickView中直接报告。实际上，是另一种方式。模型列车发生，然后ClickView将数据拉入其中。这是我们学到的与ClickView的一个很好的集成技巧，尽管我们还有一些其他问题，你不能在ClickView回传变量，因为我们的安全问题。但最终盲测精度提高了50%。所以挑战，老实说，最大的挑战是和一个你不熟悉的业务部门合作很难。 It’s super hard and you can’t control things that happen afterwards. Ownership is, I think, our biggest challenge. Even with the great handoff, which it was a great– you guys did great. You give us everything we need. Documentation was amazing. Even with that, then you get into this internal business fight of, “Hey, this doesn’t work. We thought you would fix it.” And I don’t know a whole lot about the call center, but no one else knows a whole lot about RapidMiner. So it’s this weird limbo. We’re managing that. The other piece, and doesn’t hit this one as much as our other things. I said we only have two in production. We move this week. I’ll almost say we forced this into production, made sure we got there. And then it’s sort of hands-off for me. It runs while retrains. I don’t have to worry about it. I just look at accuracy, not even weekly. Once a month I call up a call center and ask them how they think it’s doing and look at the results and stuff, so.

2008年03月08日但是我们的其他东西是基于这个的，没有人想要把东西投放到生产中，因为你会失去控制，因为我们对生产的定义是模型运行，再培训，处理自己，没有人碰它。如果你想触摸它，你需要把它从分期中拉下来，搅乱它，然后把它推回去。我们甚至不能停止生产。所以你在分期的临时版本下拉。所以人们觉得这很麻烦，所以他们就把东西放在舞台上。所以实际上你并没有促进产品的生产。它会占用我们的开发服务器的空间，但人们仍然会弄乱他们的模型，他们倾向于喜欢这种控制。所以这是我们的责任。我们在这方面还有很多工作要做。我不知道其他人是否也有这样的问题，但这可能是我们实现的最大障碍之一，就是在生产中很好地定义东西。 This is really where I want to get to, Clarkston Consultant as a catalyst. So this was our first RapidMiner project. We chose to engage a consultant because we didn’t want it to fail right out of the gate. And we wanted to see our blind spots. And we had more than I even realized. And some of them we’ve been able to take control of and fix, some of them we’re still working on. Certainly showed us what an end-to-end project needed to look like. And seeing how things need to turn out is a lot more important than trying to take things to how they should turn out. So it was really helpful for a roadmap for us. And they ask the right questions or re-ask questions that we should be asking in a way that we can make sure they’re answerable and accountable.

17:05即使这样，即使有最好的正确的问题，还是有一些问题，因为最终业务部门认为他们知道他们想要什么，直到他们得到它，然后他们意识到他们想要更多一点。所以每个人都说我们想要的是准确性。第一个问题是，“我们如何调整关键驱动因素?”就像，“你不想要关键的驱动程序。我们问你，你想(笑)在模型中加入这些变量吗?而你拒绝了。”所以人们会撒谎，忘记他们说过的话[笑声]。所以，是的，即使你问了正确的问题，也不能保证你会回答真正需要回答的问题，所以。而且，说这是克拉克斯顿的项目，而不是我的，其实很好。就算失败了，我也不用担心。 Comprehensive documentation, I am terrible at documenting my code. If anyone saw my wreck of my processes, it’d be 40 and then one statement [laughter] for the operators and one statement, whereas having a consultant, it gives you an idea of how that needs to look for people to understand it. When I got 40, I just wrapped it in a sub-process [laughter], put that one word on it. Looks great. And then gaining additional hype [inaudible], any time you bring a consultant into a project because you’re spending money on it. Eyes are on it from senior leadership, which can be really important when you’re trying to roll out something like RapidMiner and get people thinking about data science. Seems really weird. Spending a lot of money creates the hype in and of itself. So it worked well for us.

18:54然后确定基础设施的不足和知识的差距。我学了不止一个技巧，但是一个我一直在用的技巧。所以我将谈到基础设施的不足和知识差距。但我学到的一个大技巧是时间序列的挑战当你在做任何机器学习时，[听不清]树，神经网络，或任何你需要对数据进行窗口化并得到t1, t2, t3的东西。窗口，你的响应变量，在这里，我们的呼叫量，这很棘手。所以当你建立模型的时候，你不需要是t90之类的;你必须向前看。这是一个例子，这不是他们实际做的，但我称之为循环应用技巧。所以当你可以只预测t0来建立模型时，但是当你必须预测t1时，你会把数据存储起来然后使用记忆运算符，把它临时存储起来，你把它放在这个循环中，让它看起来像它需要的那样。你打开窗口，应用模型，得到下一个值，让它看起来像它需要的那样，然后把它加回去，重复多次，在这个例子中是90次。 But I have one running right now in production that’s going to 2,160 hours forward. Unfortunately, you take a performance hit when you’re doing the loop supply. So if I talk to [inaudible] [laughter] about [inaudible], I understand. And you guys are– have some better methods. So very excited about that.

20:42所以循环应用的另一个问题是你必须关闭并行执行，这对RapidMiner来说是很好的，但你会添加一个值，你在循环应用时没有之前的值，如果你试图在四个不同的过程中做它。所以你要取消选中那个方框。老实说，这才是真正减慢处理速度的原因。从克拉克斯顿发现的漏洞。老实说，我们没有，而且我们仍然——这可能是我们做得最辛苦的一件事，将我们的分析过程正式化。CRISP-DM吗?我们遵循什么方法?需要一个正式的过程是非常重要的。这包括知识转移、不参加会议，至少还有人员管理的概念。当然，我们没有真正的数据工程团队，有很多关于IT应该做数据工程的讨论。 But their concept of normalizing data is, I learned, entirely different than what I think of normalized data. Really my normalize that is tidy data. Each records an atomic unit for which I want to predict in each variables and column. They’re normalized data is everything’s indexed in the snowflake schema. That doesn’t help me. It doesn’t help me. When you say you have information on when people quit paying their bill or need to be written off and it’s just dates of the bill, and then it just stops. Well, that’s not a result. I need to turn that into one record per person. So having a data engineering team, that isn’t necessarily IT, it’s almost the pseudo data science, data engineering where they know data needs to be in this form to actually get a model to work.

22:42天气很好。斯科特·根德隆，今天早上的报道。但即使在那时，他也有人们——是什么东西转过身来还是没有转过身来?我们的数据不是这样的。这只是一堆数字，只是他们账单支付的一堆数据。我们开发了一个沙箱，我们现在有一个MemSQL用于我们的快速IO, RapidMiner能够将数据移动到列存储中，并很快地恢复数据，并且不会占用RapidMiner服务器上的服务器空间。这是非常重要的，它让我的生活变得更好。实际上有一个运行模型或维护团队模型-我们怎么称呼它们?样板办公室团队，我们真的没有。我认为这将有助于我们解决所有权问题。 But the issue there is now you’ve got to hire people to do that. And then ultimately, the biggest recommendation that we took from them is keep pushing analytics. So that’s when we started the hackathons. That’s when we started really pushing our monthly RapidMiner user groups and making sure senior leadership was getting involved with that, so.

24:02下一步行动。就像我说的，关于电话数量的预测，他们问了很多问题。下一个问题是，好吧，我们有这些。我们实际上需要通过调用类型，因为我们实际上不能正确地重新分配人员，除非我们有调用类型。然后说，你说你第一轮不需要叫type。嗯，他们做到了[笑声]。所以我们分开预测。当然会延伸到我们所有的呼叫中心因为这只是我们最大的呼叫中心。引入更多的变量是有必要的。我们正在努力，但老实说，它还在运行。 It’s being used but more it’s being used sort of in conjunction with the old. The old didn’t suddenly go away. They didn’t suddenly quit. There’s two people that didn’t suddenly quit working. They’re still there. We had no appetite to fire two people because I was there. So they’re still doing that. Maybe they’re doing some other stuff, but it’s hard to get people to let go of their old way even when this is performing better. So this ends up not really being a replacement and just maybe an augmentation. But honestly, there’s still a really manual component that people are doing because they don’t want to let go of that manual process that they do. And I can’t control that; they’re not my employees. It’s not my cost center. But that’s a it’s a tricky piece. And then ultimately. Our next steps with first with RapidMiner for [inaudible] is we use this as a good example and it gave us a lot of leverage and people are able to see how this works and get ideas, how good products work. So I think that’s all I got. Actually have notes on the slide. Oh, don’t assume your business units not lying. Yeah, big liar. [laughter] [applause] [music]