El Dorado County Part 2
I was working at El Dorado County converting their Social Services systems from an old Sperry UNIVAC mainframe computer to a new IBM one. The task had a completely unrealistic deadline and although I was seriously engaged in doing it, I was not feeling a sense of urgency since failure was a sure thing. The two main systems were the Child Services System and the larger Welfare System, that included Food Stamps and AFDC. There was no way any one person could convert those two systems within that deadline. Heck, one person wouldn't even of had a chance of converting one of them. It wasn't a normal conversion. It was essentially creating a whole new Welfare System. Did I mention that I had not previously worked on a Sperry computer before so I was going through a learning curve to boot. Regardless the conversion project was something that was beyond my control. There was no way it was going to get done on time. It was out of my control so I wasn't stressing.
A new CIO (County Information Officer) hit town, John. John had me in his office and had just informed me he was bringing in a team of 3 contractors for the Child Services System. He wanted to know what it was going to take for me to do the larger Welfare system.
“John, I need 2 or 3 more of me.”
“I can get you one right away Dave. I'll get you another in a few weeks but probably can't get a 3rd.” “I'll line up a few consultants for us to interview right away.”
Well, at least John understood the magnitude of the task at hand, but now, I actually had to accomplish this task on time for real, or at least seriously attempt to. We interviewed 3 candidates the next couple of days. One I didn't feel was really qualified. Another was Rex. Rex had decent experience looked to be a pretty decent programmer. He had less experience than me but he seemed like a good guy. The last was Bill. Bill was extremely qualified with tons of experience, but came across a little arrogant. After the interviews John and I had a conversation after which he asked me which one I wanted.
“Well, I need all the help I can so get I guess I should take Bill.” I replied.
John looked me in the eyes and asked “You are going to need to be working a lot of overtime right?”
I looked up and confessed “Yeah, probably including weekends too, at least for awhile.”
“So think about it Dave. You will be spending a lot of time with this person. Are you sure he's the one? Let me ask you again, which guy do you want?”
I looked back at John and smiled. “I guess it's going to be Rex.”
John nodded and said “Good choice Dave.”
I learned a lot working for John. He was direct. He was a bit gruff and he could be hard on you when you did something he didn't like. But he was loyal and always stood up for you. He never left you out on your own and he would do whatever he could to help you. I know many of the staff were leery and a bit intimidated by him and as a result they kept their distance but I found out quickly that being open and honest with John worked in my favor. Over the years whenever I screwed up I'd go to him and he'd work with me to resolve any problem. He totally focused on solutions not bothering about blame. Some of the others did the same. The rest felt John favored us because he would help us but he helped them too but expressed his irritation that they had kept him in the dark. He was one of the best bosses I had during my working career.
So I could now mostly focus on converting the Welfare System. You may have noticed that John was able to get 3 contractors for the smaller Child Services system. Here's the thing. In government, politics plays a larger part than in the private world. El Dorado County is not a particularly large county and there are not many layers between the elected officials and even someone like me. John in the CIO (County Information Officer) position had to deal directly with these people. John was directly tied to the CAO (County Administrative Officer). Being political, not all department heads played nice with each other. The Child Services head got along with both John and our CAO and he was willing to help fund the 3 contractors needed for their system conversion. The Welfare Department head was not a fan of the new CAO or John. He didn't trust IT at all and resented having to use any of his budget for IT support in the first place. As a consequence he refused any request for additional funding. So John had to find other funds in our (IT) budget and hence there was only funding for one contractor to help me until John and our CAO could find some additional funding for another.
Rex came to work with/for me. I had put together a plan for converting the Welfare System and we divided up the various tasks between us. We started with the data conversion programs so we'd at least have test data once we started converting the Welfare programs. Soon we were each at the office more than 10 hours a day. I started working Saturdays too. Others in the programming group were working long hours but not as much as I was, or Rex for that matter. I got to know Wilma and Linda as their cubes were next to mine and were working pretty long hours themselves. They had been working at the county for some time. They were supportive and would run interference for me with the Systems Programming and Operations staff as they were highly regarded.
The next month John was able to hire another contractor, Nick, to help us. Nick was an older guy. A decent programmer who liked to get to the office super early but his contract was smaller so I kept his hours to 40 a week. Rex worked long days similar to myself. Nick was not much of a collaborator but if you gave him a specific task he was happy to do it and worked hard at it and I was plenty glad to have his help. Nick didn't always agree with my decisions so I had to check his work more closely to make sure he actually did what I wanted the way I wanted. There were times when he would go off in his own direction. I would also learn that he was a little weak in testing his work. Still he was a big help.
Rex and I worked well together. We joked around, especially as we got tired as we had a lot of late nights. I got into a routine where I'd go to work early in the morning, get home late, eat a little something, go straight to bed, and get up a few hours later and head right back into work. This went on and on as weeks turned into months. I know it was tough on Kris as I was hardly home. I was now working like 12 hours 5 days a week, 8 hours on Saturday, and usually 4 more on Sunday. I also had a 50 minute commute from Rocklin that added to those hours. At one point Kris had a talk with me telling me she was getting worried about me. I assured her I was OK but that I needed to do this. I knew I wouldn't feel good about myself if I didn't give it my best effort and while there was still a couple of months to go it would eventually end. Working with Rex was enjoyable and frankly I don't know how I would have made it without him. The Welfare system was purely a batch system on the UNIVAC. The social workers relied on printed reports that were produced nightly to lookup existing people in the system. New applicants would be added into the system through the nightly batch processes and they wouldn't show up on the reports until the next day. The new converted system would have an online portion where new applicants could be added and existing applicants could be viewed and updated immediately. There would still be a significant batch process. I'm sure you are not interested in any more of the details as you must be more than ready to move on.
Some of my time was taken up by tracking and interacting with the 3 contractors who were converting the Child Services System that was being done at the same time. Luckily they didn't need a lot of direction from me and good news for them, due to my working long hours I was almost always there so they really never had to wait on me for information or a decision.
As the months went by Rex and I ran into considerable problems getting all the modified batch processes over to the new IBM. One of the major issues was getting the main process to run in a timely manner. It' wasn't like we had all night as the Welfare System was just one of many county systems that needed to be processed nightly. Our main process was taking over 2 hours to complete. We really needed to get that to run in the neighborhood of 30 minutes. Worse, the data conversion process and the main process were still failing off and on. Our programs that converted our data from the UNIVAC to the IBM worked cleanly at times but data issues were still showing as we tested daily with the previous days data. We kept tweaking the nightly process but it was still running about an hour as the deadline approached.
I came in early on a Monday morning with a couple of weeks to go. John saw me and said “Dave, good news. Nick has the AFDC process running in under 20 minutes.” I was skeptical. So I found Nick and had him demonstrate and sure enough it ran very quick. So I checked the data files and found there had been no updates. Turned out he had a flaw in his program where he failed to actually write out the updates. So yes it ran fast but it didn't ultimately do anything. Like I mentioned, testing wasn't Nick's strong suit. Any program will run faster if it doesn't have to write out any data. I quickly updated John as I did not want him carrying around any false hope. I decided to break the process up into smaller pieces in an attempt to create greater efficiency.
We were now starting the last week before the deadline. The batch process was running quicker but we still hadn't been able to get both the conversion process and the nightly batch process to run cleanly regularly. We also had numerous other “smaller” issues that we just did not have the time to fix. John came by my cubicle and told me he could get me another week on the deadline. I told him thanks, but no. I knew we needed more time than that. We were going to come up with problems regardless. I felt it would look even worse if we came up with problems after getting an extension on the deadline. Better to come up on time with the problems than to ask for more time just to just come up with problems anyway. I remember Nick was unhappy about my decision but getting the extra week would raise expectations and the mess was not likely to look much different to anyone other than us.
So the day came for us to go live on the IBM. Rex and I stayed all night babying the night batch process. There were numerous starts and stops with small fixes to both the data and the programs but somehow we got it to finally run through successfully. By successful I mean we got the process to complete but it still didn't do everything it was intended to do correctly. It was a little after 4 in the morning. We were a frigging mess, but somehow we got through it. There was still a lot of work to do. I sent Rex home. He said he'd be back in the afternoon. I stayed until 10;00 documenting the problems. Then I met with the Welfare supervisor, Lila to prioritize the fixes that we needed to do before then next night's run. I then went home myself returning a little before 5 to help Rex and get ready to babysit the next nights process. This cycle continued for about a week before we got those processes to run error free all on their own without causing any major issues. We did it! It was up and running on it's own. However, we did still have around 20 unresolved, not so major issues.
This was the messiest, ugliest, and most difficult (due to of the tight timeline) project of my career. This experience along with the LA experience (see blog post “GBS: What the Heck Happened in L.A. Date 5/18/21) were two of the defining times of my work life. Later in my career I would be a major player in a high profile successful project with the State of California at Caltrans that in many ways would be the highlight of my career. However, I consider the messy, riddled-with-problems Welfare System re-write/conversion at El Dorado to be my greatest work accomplishment. It wasn't pretty and looked bad to many. It was an accomplishment only a few people could really appreciate. It was the one project I was never sure I'd be able to do and I felt that way pretty much right to the end.
Thank you to Rex, Lila and John for sticking with me and getting us through it. Also thanks programmers Linda and Wilma for moral support. Thanks to Systems Programmer Tom. Lastly a big thanks to Dennis, leader of the operations staff, for allowing me full access to operator's console that is normally completely off limits to programming staff. Without that access Rex and I would never have made it through that first night.
Brings back memories, not all good -:), and congrats on getting it across the finish line and all sorted out.
ReplyDelete