2021-03-23 00:00:00 - Joint Committee on Covid-19 and Emergency Preparedness and Management
2021-03-23 00:00:00 - Joint Committee on Covid-19 and Emergency Preparedness and Management
(Part 2 of 4)
SHOW NON-ESSENTIAL DIALOGUE
Yeah,
yeah. Mm. All right. I want to thank everybody for patience and grace as we enter back to the live the live feed here, Um, I think we are are back on. So I'm going to proceed and once again, thank our panel from earlier today. In the beginning of the hearing, they were here for about two hours with us, and it was it was very informative and quite helpful to hear directly from them. So we appreciate their time and testimony today before we move on to the next panel for today's55 oversight hearing. Just want to run through a couple of housekeeping items. Um, you know, if we do have any event of any major technical difficulties at the live stream of the hearing may stop or we may have an audio issue. We will be in touch with members of the committee and the Panelists and do a reset. Um, if you know for testimony today, we've invited members of the bigger administration members of the legislature, another Panelist, Panelists and related fields to local85 health emergency preparedness. Emergency management in the vaccine rollout. Um, the committee is accepting written testimony for many members of the94 public via electronic submission via email so that email addresses joint committee dot covid Dash 19 at Mass at Emma legislature dot gov. This email address can be found under the event description on the Web page from which you are viewing this live stream today. So we're going to invite the next Panelist forward after their testimony. Any statements that they want to provide are made. We will do our best to have members of the committee asked questions, Um, as time allows of the secretary here today and we will alternate between House and Senate members of the agenda. Um, and members please use the chat function in teams to indicate if you if you do have a question, you'd like to ask, um, as as we move along. So with that, I will admit our next Panelists, and we have a panel from the Joint Committee on Advanced It and Internet and in Cybersecurity. And today we're joined by the chairs of the Joint Committee is the newly formed committee chair Senator Barry Feingold and representative Linda Dean. Campbell is the hub chair We also have here today. Secretary Curtis would the Executive Office of Tech technology services and security. And welcome to the hearing. Thank you for being here.
Thank you for having us this afternoon. It's, um it's really appreciated. And I believe I'm going to kick off Mr Chairman, if if that. If that's good for the for the committee. Uh, okay.
REP CAMPBELL - So first, I want to thank you again for inviting us. Um, I know that our two committees here have gotten off to a great start with collaboration, and I'm looking forward to that.211 Um, I think we have um, a tremendous amount of overlapping on concern and information, and it's already taken off very well as a great collaboration. So, looking forward, a lot more work to do. Um, and so, you know, looking to the issue of vaccinations, Um, you know, our committee obviously has an acute interest, Um, in working with yourselves and the administration, um, to ensure our IT is ready for, um this increased vaccination rollout, which is really almost upon us right now. Um, and we're all very thankful for that. Um, no doubt there's been some glitches253 and some challenges, right?
And so we thought it was a really good opportunity, um, to work with, um, Secretary Wood and to ask him to come to speak to all of us today. Uh, just to267 hear about some of those challenges and the progress that's been made. A great deal of progress, I think has been made. Um, And it it will be good to hear, Um, a little bit of the nuts and bolts of that. Uh, Secretary Wood for the public is a member of Governor Baker's cabinet. Um, and that is referred to as Tech Services and Security. And he brings to this challenge uh, well, over 20 years of both public and private sector experience in this field, primarily in the fields of public safety, and IT as they overlap. Um, and one of his major accomplishments recently has been, um, a multimillion dollar, um, upgrade overseeing that within our criminal justice IT systems. A big, big job, which he has been, um, an integral part of.
So when we met with Secretary Wood um, initially and326 he was very, very happy to be invited today to meet with us. Um, I asked him to touch on, you know, the website, the vaccination website. Um how we selected our providers. Um, did we have some priority for in state providers versus out state providers? Um, and how did funding come into our selection of these providers? I think this is information that the public really has a keen interest in. So I thought that that would be helpful primarily looking towards the future. Um, but, you know, just very general information about how those selections were made and why. And as we move forward and ramp up our vaccinations, um, in my mind and I think the minds of many privacy issues are going to be front and center. Um, and we're most likely going to need some kind of documentation for vaccinations. Right? And I can see a lot of privacy discussions around how we do that.
Um, it's going to affect our everyday life. Um, here,400 um, interstate travel, certainly international travel. Um, and and Senator Finegold has been doing a lot of work in this area, and he's going to speak to that, um, momentarily. He's412 got some very I think important questions that are really upon us right now, So I'm going to, um I'm just going to wrap up here again by thanking you for the collaboration obviously, in the area of IT. I think that we can merge all of our joint information and share it. And, um and it will be very productive collaboration. And and, uh, Mr. Chair, I'll turn it back over to you to, um to hear from, you know, my co-chair. Thank you.
SHOW NON-ESSENTIAL DIALOGUE
Thank you. Chair Campbell. Uh, terrifying gold.
SEN FINEGOLD - Thank you. Chairman Driscoll. I want to thank you and Chairwoman Comerford for this opportunity. I also like to thank my esteemed colleague, Chairwoman Campbell, Um, for the work that we're doing on this newly formed committee. Obviously very important stuff and, uh, clearly very important about not only now, but in the future of the Commonwealth. My name is Senator Barry Finegold and I serve as co chair of the Joint Committee on Advanced Information Technology, the Internet and Cybersecurity. For today's hearing our committee hopes to address the efficiency and accessibility of the Commonwealth's vaccination sign up website and pre registration portal. I hope that today's discussion will be forward looking.
Massachusetts needs to make sure that our online infrastructure not only meets the challenges of today, but also is prepared to tackle the issues of the future. Our vaccination system has to be proactive instead of reactive. Towards that end, I want to pose the following questions. First, what improvements can and should be made now to guarantee that appointment sign up in the systems work well when vaccine eligibility expands? I'm excited to hear that that the general public will be eligible for vaccination starting on April 19th. And I look forward to hearing some more about the administration's plans to manage this influx of traffic to the website. Secondly, how vendors and administration officials work together to make improvements to the vaccination website, And what should policymakers keep in mind going forward when developing online systems?
Already more than one million residents have enrolled through the new pre registration system with no apparent problems. I hope to learn more about how we573 can build upon the initial success of this program. And lastly and I would actually argue most importantly, how is administration thinking about vaccine verification systems such as immunity passports? We have to start looking ahead into the future when most people will have been vaccinated and the world will start to return to a new normal. And this is going to happen very soon, probably within the next 60 to 90 days. Probably by June the issue will not be about, you know, where can I get a vaccine or about how can we get people vaccinated because there will be plenty of supply. And people there is going to be an ongoing discussion about whether people should622 have to show proof of vaccination before getting on an airplane or630 attending a sporting event like Chairwoman Campbell said.
And634 people will feel more comfortable entering a public space if they know others have been vaccinated as well. For example, Israel has created an app that allows vaccinated people to access indoor venues and other countries are exploring similar systems. Likewise, in New York City, Madison Square Garden658 requires fans to display a QR code that verifies that they have been vaccinated or recently received a negative COVID test. So my question is, what are we going to do at TD Garden? What are we going to do at Fenway Park? What are we going to do with at Gillette Stadium to verify if a fan has been vaccinated? And the question is has administration discussed vaccine verification system with the Biden Administration or with the private sector. And earlier in the694 pandemic, we made the mistake of relying on the federal government to have a plan.
We can't make that same mistake again. We can't wait for federal intervention. The Commonwealth now needs to develop a system, a standard verification system. Okay. And I'd really like to know and understand how the administration is thinking about this complicated issue. And I would suggest the Legislature not only push the Baker Administration, but we also have to push the Biden Administration because in the best case scenario, there should be a federal application that displays like a TSA that everyone can get if you're vaccinated. But once again to emphasize we can't wait for the federal government. We748 made that mistake once we can't do that again. So overall, I look forward to our discussion and I'll turn over to Secretary Wood for his opening remarks. Thank you for giving me this time to say a couple words.
SHOW NON-ESSENTIAL DIALOGUE
Yeah. Thank you. Walter National. Everybody hear me. Okay. Thank you.
CURTIS WOOD - EOTSS - Uh, good afternoon Chair Comerford, Chair Driscoll, Chair Finegold and Chair Campbell. I want to thank you for inviting me today. Uh, I just want to open up with a few remarks and then certainly get into, you know, the panel session and questions. Uh, but I would say that we're very excited to see that the Legislature created this new Joint Committee on Advanced Information Technology and Cybersecurity and Internet. And in my role as a secretary of technology I've been in the role for about almost three years now, Uh, and our organization has been a fairly new organization with the Executive Office of Technology Services Security Cabinet level positions since 2017. Uh, and we've really been focused on over the last several years of modernizing the technology infrastructure of the Commonwealth of Massachusetts, as well as making sure that our cybersecurity posture and our threat management capacity is improving every day.
Uh, and with all the future of technology, future of work and technology, Internet of things, it's very exciting to see the Legislature take a very active role in this area. I've certainly had conversation with members of the legislature over the past couple of years in certain areas to talk about this. I represent the state of Massachusetts at the national level with our National Association of Chief Information Officers. Certainly been in DC talking to, you know, congressional leaders as well as, uh, you know, national leaders in this space as well. So it's very exciting, uh, to be able to have, uh, that dialogue, that conversation. So I really look forward to that. Spoke a little bit to the chairs yesterday about my enthusiasm. Uh, as Chair Campbell mentioned, I've been I've been a career government employee for my entire adult life. You know, I'm all about the Commonwealth of Massachusetts. I'm all about making sure that the residents, the visitors and, you know, business of the Commonwealth are treated with respect and get the services they deserve and demand.
Uh, I think I'd like to talk a little bit about our organization, our organization as a whole. You know, we're responsible for about 550 employees within the organization and we're responsible for our infrastructure hosting and network services from the executive branch. We also provide services to independents and other organizations as well as constitutionals. We provide enterprise and user support for our desktop, uh, desktop support for all of our agencies and organizations. That comes in a myriad of different organizations, whether it be the state police or parole, or be it Health and Human Services. It's a wide variety of end users that we support day to day. We also have responsibility for our digital and data services.929 I'll talk a little bit about that later. And also, like I said more recently, really at the enterprise security of the Commonwealth, the executive branch and cyber Security, which is which, in reality it's much more than the executive branch.
It's actually a partnership with our local government, our municipals, our national and federal partners, our other states as well. Because we're all in this together, uh, and really building out the fabric and the DNA of our organization and more importantly, the Commonwealth as we move forward in this area. We've also invested in to Chair Campbell's point, our organization actually does have a full time chief privacy officer. We have a chief data officer, chief digital officer and chief information966 security officer of968 the Commonwealth all statutory positions as part of that Article 87 reorganization. We also recently stood up an information, uh, an information governance office, along with a chief risk officer as well.
So, you know, we're more than just an IT organization at this point we're really, you know, especially the cardinal level position and executive level have a seat at the table to really make sure that we can influence the business, uh, and make better technology decisions994 and make better investments in our technology. So I just want998 folks understand that, or at least at least to recognize1001 that. And, you know, I think with the creation of this group and this committee and also with1005 the ongoing work of Legislature, especially with the investment that legislature has made with1010 capital bonds. We're one of the few states in the nation that actually has a very robust capital program when it comes to IT investments. So I want to thank you for that. Without that, we'd be in a much more difficult place, uh, than many states.
Anyway, so just talk a little bit about the COVID-19 by perspective. You know, our organization has been in it since day one again from an infrastructure services perspective and from desktop. We've partnered with all of our secretary, chief information officers and our agencies over the past year especially in the first several months of the pandemic to be able to stand up a remote hybrid workforce where we were able to rapidly based on1048 some decisions and some infrastructure we had already built out in the past couple years, really thinking about continuity of government services and resiliency. We were able to quickly stand up you know, organizations remotely where1059 our business people were able to work remotely and still continue to work remotely. Today we still at the height we had about 23,000 24,000 people, both in the executive branch and from constitutionals independents working remote.
Uh, we were able to quickly procure over 70001075 devices for people to be able to work remotely. We're certainly working on our future of work environment right now where you know the workforce has changed. The locations of the officers have changed. But overall we've had some great success, and we've not really missed a beat in providing services to the folks. And it's been a big jump to be able to manage a workforce remotely, uh, than in an office. And it's it's certainly presented certain challenges with access, and it's presented challenges from cyber security and things of this nature that we're all working1105 together to make sure that we, you know, across the T s and dot the I's. Our network and security operations teams work 24/7 in order to manage and monitor the network and our cyber security, teams are monitoring millions and millions of threats a day that are coming1118 from all over the world trying to penetrate our network and try to get into our networks.
Now we're not alone. This is happening in every state it's happening in every business across the United States and the world. So we have made some significant investments in our cyber security in this area. We continue to grow security operations, center threat intelligence, threat monitoring. We're dealing with phishing campaigns, we're dealing with, you know, stolen credentials every day. So you know, this is a real real concern of ours. And I want to thank1147 the committee and also the Legislature, especially from a funding perspective, to recognize that and to be supportive of us there. It's been it's been very, very heightened since the pandemic, with additional threats and phishing and probes into the network, especially when people work from home.
I am glad to report that we have not had any real significant issues that have happened to the state. We've certainly dealt with individual situations Uh, you know where people have been phished and things of that nature but nothing wide scale. Knock on wood I don't want to jinx this, but, you know, we continue to monitor that and manage that with the federal government with our state partners and local partners as well. Uh, some of the other areas again from the from the COVID perspective. You know our organization was able to stand up with Mass Alert alerts, a with the Massachusetts company Everbridge. Talk a little bit about that a bit about standing up, you know, a texting service that allows information to be sent to subscribers. We've sent thousands of messages over the past year, an informational on on COVID and awareness and rules and things of this nature.1212 October 4th 400,000 subscribers.
1213 Our digital team continues to partner with different state agencies, especially from a you know, the mass.gov infrastructure and the ability to make sure that we have you know, that presence on digital presence both from social media and from a mass.gov perspective to make sure that there is1230 that one stop shopping for information. We're able to stand up and support since day one a lot of the informational campaigns from the public health section, Health and Human Services, you know, MEMA, public safety, whomever you know, to be able to stand up this website or it's more than a website. mass.gov is the website of the Commonwealth of Massachusetts. But there are multiple instances within this and landing pages with information that we've been able to add, and we continually refresh, uh, in that area.
We will also be adding a chat bot the next couple of days as well for COVID-19. So that helps with questioning things of this nature as well. So we continue to evolve the mass.gov site in order to1266 be able to support and provide information and hopefully, you know, making an easier experience and to have, you know, a central location where people can get reliable information and have so to speak one truth of you know one sense of truth. Uh, in February, of course Uh, you know, when the provider websites have the issues, you know, the PrepMod the Collar and the Curative websites that were deployed to support or deployed to make the appointments when we had that, you know, challenge in February when we got the Orange Octopus and we got the, you know, we got the issues with booking and the people trying to get appointments,1307 things of that nature.
Based on this unprecedented volume, I don't think anybody ever expected, you know, again, you know, without having enough vaccines, you know, trying to compete for those appointments was certainly a challenge. What I would tell you is that our organization, the Executives Office of Technology and Services immediately, you know, jumped in there1326 with the three1327 vendors PrepMod, uh, Collar and Curative. And along with the Project Beacon team, who had built out that scraper tool in order to provide a central location of pointers system where available vaccines were. Our team was monitoring from the Digital Services Team, our infrastructure network team. We immediately jumped into action with those folks, and we're able to mitigate most of those issues within a week or so.
And some of the areas that we really focused on and certainly take some questions on this you know, after my testimony, of course as well is, uh, you know, when we talk about we brought in a Akamai, which is Cambridge based company. Akamai is probably the best well known content delivery internet company in the world. We brought them in and stood up what we call a content delivery network in front of the, uh, in front of the Project Beacon Scraper to allow us to do a better job of managing, uh, managing the network, managing the load on that application of the pages. I will say this that mass.gov itself did1383 not experience any issues throughout the entire month of February or since. I mean, the mass.gov infrastructure is very robust, it's very resilient. It doesn't miss a beat.1392
Uh, some of the experiences that were felt by the citizens and the residents and the folks that try and get appointments for specifics of the applications. Uh, we did we deployed, uh, you know, we immediately contacted and worked with these three vendors to make sure that, you know, we took some additional steps, they took some additional steps. We immediately demanded they put in additional technology appointment, features with waiting room features. As you also you saw. We certainly had some challenges the first week of that as well. But what I would say is that you know, our team worked diligently with them to make sure that we didn't have a repeat of that February 18th situation. So between February 18th and February 24th, we had a few challenges. But for the most part, you know, everybody was able to make appointments they got in the system.
At the end of the day, the real challenge was, you know, hundreds of thousands of people trying to compete for a very minimal amount of appointments. And I think we all1444 recognize that. Uh, and parallel to that we had, you know, we had made the decision to move forward with a pre registration system. So we were able to engage Google in a very quick time. Within 10 days of engage in1457 Google, we were able to stand up the preregistration website with Google. And on the first day, with first 24 hours, we were able to register over 500,000 people. As the chair mentioned earlier as of today, we have over a million 0.1, almost a million 0.2 of folks that are pre registered without incident without missing a beat. There's been no performance issues whatsoever with the Google infrastructure and I can talk a little bit about that in a few minutes.
We believe we certainly believe the pre1488 registration system is ready to go. There will be no issues when we added eligibility when the folks on April 5th and April 19th we don't anticipate any problems. I'll give you a sense of scale. The first day that we were live on March 12th with the Google infrastructure and for the folks that know a little bit about Google. Certainly I mean, we all know who Google is, but I think from a technology perspective and the infrastructure, they have more of the resource than the actual Internet just to give it a scale.1516 Uh, and the way they manage is, uh is they're instances just spin up automatically when they see a certain threshold of people trying to enter or trying to transact their infrastructure automatically scales up.
So that very first day, you know, we we had started off with about four instances and moved into, like, 14 at the peak, and we were managing hundreds of hundreds of submissions a second without missing a beat. And as the as the infrastructure scaled up, the performance actually improved, of course. And then the wait time actually went down. So all my point being is the Google infrastructure from the cloud services and network and infrastructure, you know, we do not anticipate any issues at all. Uh, you know, we've since had eligibility, uh, this past week increase. We have another one next week on the on the 5th and another one of course, the 19. We don't anticipate any challenges at all from from a network or going into the pre registration system.
SHOW NON-ESSENTIAL DIALOGUE
Uh, just, uh, I'm good.
Thank you. Sure. Mhm. Okay, Mr Chair, you may be frozen. The, uh
yeah.
All right. Yeah. The Prime Minister, Secretary. Some of the members questions we're halfway through. Buy time for the Okay. Okay. I'd like to, uh,
the first question,
Mr. Chair, you were frozen and I was unable to hear you, so I'm gonna let you. I'm gonna let you do the tech unfreeze. Um, and then I'll just call on Senator Lesser. Uh, and then we'll go back to you. I hope that's okay. Please, Senator. Lesser. Thank you, Mr Secretary.
Uh, thank you, madam Chair. Can everybody hear me? Okay. And everybody can see me. Okay, we are good to go. All right.
SEN LESSER - Uh, so thank you so much, Secretary Wood, I appreciate the testimony. And just to kind of summarize what I think we heard you say, um, you described yourself as having quote a seat at the table for the Commonwealth's IT Infrastructure and procurement. Is that that's that's safe to say, your agency is the kind of Chief IT uh, information tech procurement and overseer1673 and coordinator for state government.
WOOD- That's correct.
LESSER - And you also talked with some detail about actions you took subsequent to1683 the February 18th website crashing including working on the preregistration improvements, including working with the, um, with the three vendors that you mentioned. So I want to just ask before February 18th did you personally or your staff have interface with HHS in procuring the website to start?1704 I understand you got involved after the crash, but I'm specifically asking before February 18th.
WOOD - So if you're if you're referring to the website and I want to, I just want to be clear because, you know, it's not just one website, so I think we need to understand the level site. And I think I think we all know this, Uh, we're really talking about multiple appointment providers being Color, PrepMod and Curative. They were contracted by HHS to schedule appointments with the labs uh, to do that. So that's separate those those are not does not Commonwealth websites. The Commonwealth website is mass.gov.
LESSER - But you yourself just said HHS contracted with those vendors to do that booking. So did you have conversations with Secretary Sudders and or did you coordinate with your counterparts at HHS about that procurement?
WOOD - I did not.
LESSER - Did you do so? Did you do a load test, or are you aware of a load test being done before the February 18th eligibility expansion on the website's ability to absorb a million new eligible appointments?
WOOD - So let me begin. I think we're mixing apples and oranges respectfully, Senator. So the million new is really from a pre Reg I think we're talking about.
LESSER - No, no, no. On February 18th, the eligibility expanded it and the governor had announced on a schedule that's 65 and older, and those with certain medical conditions would be able to pre register on the site that I believe you just mentioned HHS had contracted1805 to, and that was the day it crashed. So what I'm asking is, is the day before that or the week before that or the month before that was a load test on or any kind of capacity analysis done with the vendor about the ability to absorb those new appointments.
WOOD - So again, not to be too argumentative but I just want to make sure what's talking about the same issue. It's not, they were not pre registering. So those appointments were being made directly to the three specific vendors PrepMod, Color and Curative. When appointments became available when they get announced, people go and try to make appointments within those respective websites. I did not personally load test. I am, I am. I believe there was testing done with the vendors. I believe there was no reason to believe that they would fail.1857 Uh, it's evident from what happened they did. So whatever testing was done certainly was not, uh, successful enough for these systems to not perform as they were expected.
I would say that again the real issue there is, uh, you know, with so many people trying to get down a small pipe, so to speak, in order to compete for those appointments is the real challenge. Uh, and that's when you know, we stepped in on the 18th and we worked with the vendors to deal with several issues again, from a traffic control1895 perspective on the front end. Realizing at the1897 end of the day, we all recognize those systems on day one, specifically one PrepMod failed miserably. We all agree with that. The people's experience was terrible. People were upset. They should be upset. We were upset. Uh, we saw what happened unacceptable. Our team went into action with those folks with directly with PrepMod technical people directly with the Color people and with the Curative people.
We said this is not acceptable. We made some direct requirements or recommendations that were really more than a recommendation they were directive made personally to say We need to do this differently. Same with the Akamai with the Project Beacon that when the Yellow Octopus Orange Octopus showed up, we said That's unacceptable as well. We immediately put some new technology in there, and we procured through Akamai from the front end to the back fighter to be able to do a better job of content delivery so things wouldn't crash. And also worked with those three other vendors to make sure that they immediately that week went in and put in a virtual waiting room. They put in appointment lock and more importantly, they put in the content delivery network in front of1967 it.
LESSER - So, uh, secretary, I'll yield back Madam chair. But I just want to confirm, because this is very important for us understanding moving forward how we get the pieces of government working in the same direction. I'm correct in understanding that you and your team did not participate in the hands on engagement with those vendors until February 18th. Moving forward. Correct.
WOOD - I would say we were involved with them before February 18th, but not selecting them as the vendor to do the actual appointments. But we were involved we were in discussions with them from a planning perspective. From a rollout perspective, we did have a conversation with him. We went through a checklist with them, you know,
LESSER - So if they check if you went through a checklist was a was a load, you said a load analysis wasn't done, though. Is that a part of the checklist?
WOOD - They assured us that they had met. They assured us that they, they assured us that their network was sufficient. And they had met all the requirements to do that. We had personally, personally did not do a low test. TSS did not do a low test. They assured us that they were able to handle this traffic.
SHOW NON-ESSENTIAL DIALOGUE
Um, Senator Lester. Thank you, Mr Secretary. Thank you. Um, uh, I believe that Chair Driscoll is still having Internet difficulties, Which is ironic, given the subject of our our hearing right now, I'm gonna intervene and just welcome Representative Dom,
Representative, we can't hear you. You can hear me now. I think. Thank you, buddy.
REP DOMB - Thank you, Madam Chairwoman. Thank you, Mr. Chair. Um,2062 and thank you very much, Mr. Secretary. I have a couple of questions and some of them are riding on what Senator Lesser's questions were. When I2070 was looking at the contracts that the committee had been able to obtain2075 from the administration it looked like there's a flurry of contracts related to the website starting around February, moving forward. And before2084 that, there's one contract for PrepMod in August for $125,031. I'm not really. I guess that was for the original website. And so my question is also about, but I just want to clarify based on your answer, that you were in conversation with PrepMod, but you weren't really sort of deeply involved with developing the website until things went south. And then it looks like you came in and sort of righted the ship. Is that correct?
WOOD - Well, I would say I would say this, uh, our whole our mass.gov team and our2121 technical team has been involved, you know, with the whole COVID preparedness from a mass.gov and working and our architects and our folks work on them. HHS did make the procurement directly with the PrepMod folks and the other two vendors again, based on they had provided the service to other states. They provide the service. Their appointment feature is just part of their, uh, solution. So that's not uncommon. And I just want to clear up something to Senator Lesser, and I get I get his point. Uh, but I also want to, you know, I'm certainly responsible. I am. I'm the secretary of technology. So I have I have a statutory authority of all technologies State of Massachusetts, of the executive branch. I take full responsibility for that.
But we also2160 have a delegated model and especially from the business side of things from an application perspective. We manage infrastructure, network things of this nature. So2167 we do have oversight, but we do have delegated authority with the business and the local SIEO, secretary of Chief Information officer, work together under our framework in order to deploy these business applications. We consider this as a business applications and extension called a website certainly an appointment scheduler but it was not specifically part of our mass.gov enterprise. So getting back to your question, ma'am. Uh, so we were certainly working with them previous to the 18th. You know, we had met the folks in PrepMod and Color and Curative.
We had worked with the technical folks, Uh, you know, to make sure that people were prepped and ready. We understood what was going. We had designees assigned to the project. We have folks assigned to the Command Center, so it's not like we weren't2210 aware of this. We were totally aware of this, but that we felt we felt there was no need for us to be able to be in the weeds at that moment because there was no indication this would fail. That's the bottom line. When it failed on February 18th, I personally jumped in and took over you know, the really the management of those three companies, from from a technical perspective, not not the contract, but from getting this thing fixed and getting it straightened out so that we did not experience another situation that we experienced on the 18th.2240
So our team dug in. We made some technical assessment very quick technical assessments. We worked with some of the vendors and we put a program together, a plan together a very quick order to to re mediate that so these things didn't happen. We did have a couple of bumps and bruises during that week between the 18, the 18th and 24th. We recognize that, but we're still able to make appointments. Fundamentally, we still had the biggest challenge we still have was the 200 you know, for a minimal amount of appointments with all those people coming in the front door. And that was the biggest challenge we had, not only from a crash perspective but also the user experience. People sitting there for hours trying to get an appointment which we all know is not the way to do it.
DOMB - I understand. But you know, I do go back to something I said in the prior, not today's hearing, but an earlier hearing that2285 we could have. We could have projected how many people would be interested in this because people have been in their homes for a year. We know by the census tracks how many people are 75 years and older. We had projections around, so there could have been a way not necessarily for you but for other people to project. I just want to pick up on one more thing that you said when you, um about how it wasn't part of our sort of, um, it wasn't part of our plan or it wasn't part of our I forget, the word that you used our enterprise.
And I think that that is key. I have to say that I'm feeling that until it2322 crashed, it might not have been a priority for the administration. Because October 7th we see that there's a vaccine advisory committee established to create the phases of the rollout. October 16th the state submits a report to CDC with a plan which I read. And nowhere in this plan is there any even mention of a scheduling website as to what we're going to do, even though it comes maybe two months after PrepMod's contract.2351 So I'm not faulting you. I think that distribution was a priority, but maybe access not so much until it crashed. And then it was like, you know, light bulbs over the head. We have an access problem. Um, and I don't blame you at all. I'm grateful that you came in in February and fixed it to tell you the truth. And I love the Google piece about how it expands with demand, because we're going to need that on April 19th. Do you envision the pre registration process being open to more than just the mass vaccination sites?
WOOD - Uh, that's under discussion right now. Now, what I would say, that's more of a policy perspective. But at least from my, uh, you know, my understanding, I don't think we would see the CVS as the hospital networks be part of that. That's certainly run separately, but I think we are having some discussion about regional collaborative and other opportunities. Uh, so I would say we2406 do hope to add some sites, But again, that's really going to be a policy for Secretary Sudders . We're ready. We're ready. We're ready. From a technology perspective, we are ready to support whatever administration would like us to do in this area. I feel we're at a very good spot right now technology wise.
DOMB - So the existing pre registration process could accommodate the regional collaborative.
WOOD - Absolutely, absolutely.
SHOW NON-ESSENTIAL DIALOGUE
so much, Mr Secretary. Thank you. Thank you. Thank you, Representative. Dom, I believe that Chair Driscoll is still having tech difficulties. So I'm going to step in Mr Secretary with a question on behalf of the Senate.
SEN COMERFORD - Um Representative Domb started to get a line of thinking and a series of questions that I've had as well. And that goes to the preparedness issue. Um, and it goes to something you said, Mr. Secretary, that we we couldn't have anticipated the demand. Um, and yet from doing our own research in preparation for this hearing, you know, I've come to understand that there are major schools of thought right2466 here in the Commonwealth that actually marry supply side concerns with IT with tech, and they come up with solutions tech solutions that really help manage demand and supply. Um, so my question is actually a two part question. I'll ask the first one first. Um, did you,2486 did the Commonwealth, to your knowledge, reach out to the big brains here in the Commonwealth and say, We've got limited supply and we've got millions of people and we have to build a tech interface that's going to help maximize access, um, and speed at the same time. So equity and speed. Did you Did you have those preliminary conversations when you were setting up the different sites that would be accessible to residents?
WOOD - Ah, I did not. I can't I can't speak for2516 Secretary Sudders, but I would like to, you know, correct one thing I said and be clear. When I talked about the man, I was all the other half of that I think you brought up. I think the other half of that, I think, you know, it's fair to say that, you know, if we would have had more vaccines available, this might not be an issue. I'm not I'm not dismissing the fact that we had some website issues, but it was really based on, you know, almost the panic of people trying to get there and get a get an appointment, right? If the vaccines were available that it might be saying it might, it might have been a different story. I'm not specifically
COMERFORD - Mr. Secretary respectfully to you, sir. If we had, you know, if we had let folks know that we had a million appointments available in one day, we might have had four million people try to access those millions of appointments so it could have cut the other way. Um, let me just2568 get on to my second question mindful of your time, sir. Um, so I'm thinking about Olivia Adams, Um, famed, um, tech developer who as, you know, caring for a baby and thought to herself. Boy, I'd really like to have a site scraper, because, boy, I can't figure out how to get to all these different sites. Uh, and I'm thinking about your own example of Akamai being brought in after the crash to do, um, you know, extra load balancing and support. So I guess I'm just And I'm thinking about the2602 waiting room, right? And the preregistration, I guess I'm just wondering, these are pretty basic best practices, right? Not not advanced tech. And I'm wondering why we didn't do it to begin with all of these things. You know why? What? I mean, I'm grateful to Olivia Adams for showing us the way. I'm sure every constituent in the Commonwealth is. But why didn't we do it first?
WOOD - Well, at this point, I can't answer you that, uh, to be fair. What I can tell you is that, um you know, we should have We didn't and we did. But that's as far as I can get. I respect your question, I probably can't answer appropriately, but2644 you know, I think the point being is that I think, you know, we recovered fairly quickly. Uh, we solved it. It's not perfect, but we're better off today than we are last week, two weeks ago, three weeks ago. And we're getting better each day. And I think the opportunity is there for people to be able to book appointments, is there, And I think the big thing, you know, from a pre Reg I know folks here to talked2664 about this in the past that, you know, it certainly gives us it gives people a sense of, you know, they calm somewhat they know they get an appointment. They're in there, and the system is working. Uh, I apologize. You know, uh, you should've done it sooner. We didn't. It's done now and will continue to grow and improve. I appreciate, I appreciate your point.
SHOW NON-ESSENTIAL DIALOGUE
Mr2684 Secretary. We're not looking for your apology. Really? This is This is to make us stronger as a commonwealth. Um, I see my my chair here, but I don't think he's really here because he's still frozen. So I'm going to yield my time to representative jicama and forgive me, Representative I think in the2701 queue earlier, and I didn't recognize that. You're welcome now.
REP DYKEMA - Thank you, Senator. And thank you, Mr. Chairman as well. I appreciate the opportunity to be here, Mr. Secretary, and to have a chance to ask a question. And this has been very informative to sort of hear the process that we've gone through. And one thing that's clear, um, to me and probably everyone is the role that2723 technology is playing in the successful vaccination program and successful pandemic response. And, you know, we're very grateful for the new pre registration system, which has obviously I've been working quite well so far. One of my questions is, given, given the clear importance of technology. And I'm looking at the number several contracts here which have been, you know, taken by the Commonwealth over the last several months. And it sounds like several of them have been under your watch and with your guidance, the most recent ones, including Akamai and and Google Cloud.
And I guess my question is really around process. And we all recognize that we're under very unusual pandemic circumstances here. And I think a normal procurement for technology of this magnitude and scope would normally take a very long period of time, which we did not have and still do not have. My question is around the bid process specifically and how were these, um, you know, contracts determined? How were these vendors decided upon what criteria was included and how extensive was that process in terms of reaching out to all the possible players and2793 contributors that might have lended guidance to this process.
And then in the same note, with respect to some of Senator Finegold's comments at the beginning about the future of use of technology, including immunization, passports and things like that. What can we expect as the Legislature in terms of bid processes for these future vendors that we're going to need? Um,2816 will there be an RFP issued, when, um, you know, when are those processes kicked off? How do we start that? And how can we make sure that we are very much in the loop and informed when these things happen? Thank you.
WOOD - Certainly. Thank you. Great question. So let me let me just let me start with the procurement under my signature. So again, you know, when this really from you know in February when things started to, uh, you know, uh, not perform as we had hoped Uh, and we stepped in Uh, Akamai, let me be very clear all the procurement that we made that I made were emergency procurement under the authority emergency procurement. However we leveraged state contracts. Uh, we Akamai first of all, let me speak of Akamai. Akamai of2869 course is a Cambridge based company that really is probably the largest presence in the world on Internet management and from content management and distribution network and, you know, has a great capacity, local resources.
They stepped up. They called us, actually, and we worked with them and we executed a contract very quickly with them through an emergency procurement process. Uh, you know, they're local. They're well known. Our technology folks from our organization, you know, we've used them in the past, you know, we know the product we needed from a content delivery network and also some of the other products that have, like, a wait room. So we felt they were local. We could, you know, engage with them fairly quickly and come up with an immediate solution. We had to solve this in a day, you know, I mean, relatively speaking, So I feel very good about that. There were no issues from my perspective. We picked2919 the right company. It is that it's a leader leading company in the world of this. When we talk about Google, let me give a little background on the Google.
Google has been Google, by the way, has a large presence in Massachusetts. They have over 2000 folks here. They've been meeting with us2936 over the last, you know, year or so trying to engage with the Commonwealth. Right now, we do a lot of work with Amazon or AWS and Microsoft Azure from a Cloud Services. But we've been looking at Google for quite a while. And when this all hit, we had already had Google and having conversations and we knew what they would do in other states. Our team, in parallel, was looking at other states. I2956 participate at the national level and by the way, most states, if not all states, have had very similar problems that we've had. I want to be very clear on2962 that. Nobody has had a perfect run at2964 this, even when we think they do they really don't. Uh, so, you know, from from from this preregistration system, we had conversations. Just so you know, we had a conversation with Everbridge which is a Burlington based company that we already employed for our alerting. And we met with Microsoft, who also deployed in several other states as well in this in this space.
So we looked at three. We looked at three big vendors here that actually could we knew that had the potential, actually, two big vendors and one medium sized vendor that had the potential could actually do something at a relative important time. So we worked with those folks. We2997 met with them and we came to the determination that Google was our best option at this point because of what they've done in Virginia, what they did in Oregon, what they've done in other states to set up both an end to end registration system and an appointment system, but also a pre registration system. Microsoft was kind of too big. Microsoft really was not ready to come in and do3015 it the way we wanted to do it And Everbridge wanted just do it the way they want to do it.
So Google actually was the only one that actually met us halfway and, you know, kind of manage what we had to manage. We could not in the in the in the middle of this situation, we could not throw out PrepMod Color or Curative. Not an option. We had to find a way to make eliminate them from the front end and put something in the front end to manage the traffic of the front door. So Google was the best option. Google cloud the infrastructure, so we felt very comfortable3044 with that. I made the decision on that procurement as well. Uh, in any and some of the other products that went with it are part of the Google suite. Some of the independent contracts, like Twilio, is a messaging platform. So this is just what you3056 know, we worked with Google.
So I feel very comfortable and confident, You know, we made the right procurement under the right rules. Now, to the bigger question, What I would say to you is that you know, our organization spends hundreds of millions of dollars a year in technology in the commonwealth. Uh we always, you know, under normal circumstances, go out for competitive bids. We use the state contracts. You know, we follow the state procurement guidelines to a tee. So we've had to make some emergency procurement. These are some that are the most representative of that over the past month. As far as, uh, Senator Finegold's, uh, discussion that we've had, I would say this. Certainly This is a policy discussion, I think from a from a digital passport, digital vaccine, passport, some type of credentialing.
Uh, my office is following this from a national perspective. We're following it from an international perspective.3108 We've not had any really specific, uh, directions to do anything yet. I think you know, you'll have to, you know, chat with Secretary Sudders on that from a policy perspective. I think from a technology perspective, certainly everything is doable. What I would say, though, these types of products, you know, are not without challenges. I think, uh, Rep Campbell mentioned one. Privacy is a huge issue with this standardization in order to3131 be able to, you know, work across a3133 national landscape. I know there's a lot of stuff going on in Israel with the Green Pass, and Denmark's doing some things in England. But it's, I think it's it's something that we just don't want to rush into.
I certainly respect and understand the desire and the need, but I think that's an area that we really need to think about and, you know, fromaA from a state perspective, a regional national, how this is all going to work, you know, from a transportation perspective, International Association. Air Travel is working on these. Everybody's kind of doing their own thing. So we need to be a little bit careful, a little bit cautious here, and I respect the Senator's point about Don't wait for the federal government, but we need to make sure that3167 we have the appropriate policy. That's not my area for this. But any technology investment we're going to make you know again, it's kind of it's kind of what we're all talking about here, right? Making sure we're making sure we're doing our due diligence. Make sure we pick on the right product.
So I think with that respectfully, we also have to make sure that we have the right approach and we work on this from a standard space. So I think that that's kind of where we're at on that, hopefully I have addressed that those questions And I'm certainly. And I know I'm not sure if I've talked all of Rep Campbell's requests either from a from a vendor perspective. But, you know, we work collaboratively, and I would say this as well from a Massachusetts company perspective. During this event, we've worked with Akamai3208 Everbridge, uh, Go Info, which is which is a small niche boutique company here in Massachusetts that does user experience. And I will say this to the group.3217 You know, we're continuing to refine the user experience for all of our applications, including the pre reg. We want to make sure it's the best experience for all of3226 our people, uh, from accessibility, access all those areas, plus just the utility.
I would also say the EOTSS my organization, you know, is a large organization has a large footprint and mission. I will tell you that we are very committed to making sure that we leverage small local vendors in Massachusetts as which we do. In the IT space we use a lot of women owned business minority owned business. We use a lot of staff augmentation contract as well. So a lot of our work that we do is not done by the big outfits from out of state. It's actually done by a lot of the smaller companies here in Massachusetts, and I'm very proud of that record.
We continue to do that because we do believe that we have talent in Massachusetts, not just the big companies we have smaller niche company's here to do artificial intelligence.3266 They do, you know, automation. They do. They do, you know, project3270 management. So there's room for everybody in our organization, and that's my commitment to continue that, Uh, and I'm very proud of3275 that record. I've done it for years and will continue to do this. I want you. It's separate from COVID but I want you to understand our organization is committed to that as well.
SHOW NON-ESSENTIAL DIALOGUE
Thank you, Mr Secretary. Thank you, Representative. Um, Mr Secretary, mindful of your time. And it's nearly three o'clock, and actually, we want to do a sound check for chair. Driscoll, we're gonna thank you so much for being here. We, of course, welcome All written testimony. Um, and any other kind of, you know, intelligence. You want to send to the committee to make a smarter about the work you do and what you hope to do going forward. And we're very grateful for your service. And thank you for the opportunity. I look forward to working with the committee and and the new committee as well. And if I can be helpful or if there's any follow ups, please don't hesitate to, uh, get you anything you need. Thank you so much, Secretary. Thank3328 you. Chairs. Grateful. Thank you so much. Thank you. Take good care committee committee. At3334 this moment, you are welcome to turn off3336 your cameras and mute. We are going to get Chair Driscoll3339 back in the saddle as it were, um, and through some tech difficulties and thank you to the chair for his good humor. Um, And then we will be glad to welcome Governor Baker.
© InstaTrac 2025