Your mission: Collect data from individuals on-demand, then quickly turnaround a 360 report for each person…and do it for 3,000+ people worldwide.
When individual workers and leaders are ready to solicit diagnostic feedback like a 360-degree assessment report, it’s important to strike while the iron is hot, while their attention is focused on self-improvement. Timing and speed are critical success factors.
In response to that need, the Prestera FX People Analytics team has developed a Reports-on-Demand (ROD) workflow methodology that enables individual participants to initiate the process anytime, anywhere, and have a full report sent to them as soon as all of the data gets collected. This case study describes how we went about automating the process of registering new participants, gathering their roster of 360 respondents, collecting survey data from all of them, analyzing the data, publishing reports, distributing them, and scheduling readouts.
The Challenges
Our client, the head of Enterprise Learning & Development at a global pharmaceutical manufacturer, wanted to implement two diagnostic reports that would help bring about greater self-awareness: (1) a self-assessment for front-line workers related to their organization’s core values, and (2) a 360 assessment for their leaders that focused on leadership competencies, traits, and behavior patterns. Our team had developed many such reports, but normally these reports were run in batches, once a month, once a quarter, or once a year, depending on the client’s needs. In this case, our client challenged us to come up with a different approach that would address the following:
#1: On-Demand Access
Through its natural rhythms of performance appraisals, goal setting, mid-year reviews, talent reviews, succession planning, and ongoing coaching feedback, the workplace creates windows of opportunity when individual workers and leaders are more receptive to feedback and guidance on how to get better. This window of opportunity is different for each individual based on what is happening with their professional life, their geography, and a number of other factors. So, any diagnostic reports would need to be accessible to participants on demand at any time, 24/7/365.
#2: Automated Workflow
The hectic pace within most businesses means that these windows of opportunity tend to have short life spans, and it doesn’t take long before individuals go from “I want to get out of my rut” to “Not now, I’m too busy to focus on development.” So, any sort of diagnostic tool used in career development needs to deliver a report quickly, or it risks missing the window. To achieve the necessary speed for an on-demand report, we would need to automate our process, treating each report as a batch of one.
#3: High Impact
Once they receive their diagnostic report, individuals need to be able to read and interpret it quickly, draw valuable insights from it, and turn those insights into a set of next steps, a plan that will get them from where they are to where they want to be. It needs to shed light on and disrupt their normal patterns to help them get unstuck and out of their current rut. So, any diagnostic report needs to be worth the wait and deliver high impact. Fortunately, our reports were already optimized for impact.
The Solution Architecture
Step 1: Participant Registration
Enable an on-demand intake process with guardrails to make sure the right individuals are accepted as Participants.
Automating the registration process was tricky because of the need to interact with participants from all over the world at any time of day, while educating them and guiding them through our process.
Our challenge was to enable individuals to self-register, while at the same time maintaining guardrails to ensure that only authorized individuals were able to proceed. For example, we needed to make sure that the Participant was an actual employee of the company and that they were registering for the right report.
INTAKE FORM. Would-be Participants would start their journey by clicking on a universal link that was available to them through the company’s intranet. There were two links: one for the core values self-assessment available to all employees and a second link for the 360 assessment available to all leaders above a certain level.
When an individual clicked the link, they were sent to an intake form, where they registered by entering their name, email address, phone number, and other information.
Only individuals with an active, valid company email address would be registered as Participants.
examples ROD Intake Form
APPROVAL LOOP. Since the 360 assessment was only intended for leaders above a certain level who were deemed ready for a 360, we created an approval loop where a sponsoring HR Business Partner would need to approve the individual in order for them to be registered as a 360 Participant.
When filling out the Intake Form, the individual selected their HRBP Sponsor, and so we sent that HR leader an email requesting permission to enroll the Participant in the 360 assessment.
This approval loop created drag on the process because HRBPs did not always respond immediately and often had to be sent multiple reminders.
Flow Chart for Intake Process
Step 2: Roster Collection
Help each Participant identify the right coworkers to invite to respond to their 360 survey.
Once an individual is approved as a 360 Participant, the automated workflow would send them a custom link to their very own Roster Form, where they would be asked to enter the names and emails of their 360 Respondents, placing them into their appropriate groupings: Manager, Customer, Colleague, and Employee.
WATCH OUT FOR ROSTER SELECTION! Roster selection is a key failure point in a 360 assessment because many Participants will only select coworkers who they feel will be complimentary of them, their work friends, and the people they get along with the best. They often leave out coworkers who have been critical of them in the past, who they may perhaps not get along with as well, and who may not share their perspectives and background. Often, they lack diversity, too. These biased, or “stacked,” Rosters generally produce sycophantic results that fail to provide compelling insights that disrupt the Participant’s thinking, decision-making, and behavior patterns.
EDUCATION. Prior to launching the 360 program, our team helped the client conduct a series of webinars and in-person meetings where she educated her leadership team on the 360 process and how to ensure its success. Among the key success factors was the manager’s involvement in helping each Participant select the right 360 Respondents. They were taught to ask probing questions and to challenge the Participant to seek out diverse and divergent perspectives.
CULTURE OF FEEDBACK. The 360 program also worked hand-in-hand with other initiatives related to promoting a culture of soliciting and giving honest but constructive feedback. For example, a best practice is to talk to each Respondent ahead of time to let them know what you are trying to achieve in terms of professional growth, to explicitly give them permission to be candid without fear of retribution.
TRANSPARENCY. Through our Information Guides and the Data Privacy Policy statement that each survey respondent had to read and sign, we shared how data would and would not be used, making clear how their responses would be anonymized and grouped with others to protect them. These ground rules were agreed upon with leadership and coded directly into our automated process.
examples pages from Info Guide for Respondents
Step 3: Survey Administration
Launch each Participant’s 360 survey to different respondent groups, monitor data collection, troubleshoot, send reminders, and provide updates.
Automating 360 survey administration was the most complicated part of the project. For the self-assessment, it was relatively easy because for each Participant, there was only one survey form, and that form could be filled out immediately after Registration. However, the 360 survey was much more complex and difficult to automate.
ONE PARTICIPANT, MANY SURVEY FORMS. The 360 survey involved sending forms to as many as 32 different people who were all assessing the one Participant. Moreover, each of the five Respondent Groups had its own variant of the survey form that varied slightly in wording and structure. Our automated survey management coding would need to keep all of that complexity straight and ensure that all of the data was correctly associated with the one Participant.
REMINDERS. After the initial launch of the 360 survey emails, each containing a custom link to the correct form, Respondents were given about 2 weeks to respond. Along the way, the automated system sent reminders to anyone who had not yet submitted their response. We spaced out the reminders so as not to be overly pushy, but increased the frequency and urgency as we approached the anticipated survey close date.
TROUBLESHOOTING. Since this audience required white-glove service, we provided them with human-on-the-loop support in case they encountered technical difficulties. We had a shared inbox that was checked frequently by our Survey Managers, so that they could intervene quickly and help the Respondent get their submission completed. In the future, this could be done by an AI Agent, but we didn’t feel that the audience was ready for that at the time.
Response Process and Communication Workflow
Step 4: Data Pipelining
Anonymize data, process it, and move it into our Data Warehouse, so that the data is ready for use in analysis.
We tend to underestimate what is required to turn raw source data into clean, filtered data that is ready for use in analysis and reporting. In an automated workflow, this is known as an Extract-Transform-Load, or ETL Pipeline.
Our Technical Lead’s and Data Engineers’ challenge was to:
- Extract each Participant’s data set, which could include 15-32 related data records, moving it from our Survey Management System (SMS) into our Data Warehouse(DWH) as soon as each Participant’s survey closes.
- Anonymize the data along the way to separate all Personally Identifiable Information (PII), such as their names and email addresses, from the responses.
- Transform the raw, anonymized data by converting responses into numerical values, calculating weighted scores, and applying rules for handling edge cases like respondents opting out of a section.
- Load the transformed data into our Data Warehouse (DWH) from which it could be queried by the code that handles data analysis and report generation.
Automation Flow for Data Processing
This process was triggered by an automated workflow developed in a software application called n8n. Once the transformed data was loaded into the DWH, n8n triggered a series of code packages to do the rest.
Technologies involved in the Report Automation Flow
The remaining steps in the process moved quickly. Within minutes of the survey close, the transformed data was loaded into the Data Warehouse (DWH), and then the n8n automated workflow triggered a sequence of steps that yielded a finished report.
In our GitHub code repository, we staged a series of code packages, or scripts, that handled different aspects of the process. The first script queried the DWH for the transformed data, then the next ran the analysis to calculate the metrics to be used in the report. Then a different script generated the charts and tables, while another script placed those plots into a report template to produce the report.
Step 5: Data Analysis
Run analysis on the data to yield insightful metrics to be used in individual reports.
Step 6:Report Generation
Use those metrics to produce intuitive visualization graphs and tables and incorporate them into the report.
Step 7: Report Distribution
Email the report to the Participant as soon as it is ready with a link to schedule their readout session.
Step 8: Readout Scheduling
Enable Participants to select a date and time that works for their schedule, match them up with a readout coach, and send a meeting invitation out to both.
Automation Flow for Report Production & Distribution
Design Principles at Work
This case study illustrates how the Prestera FX team is able to develop custom solutions that address a client’s changing needs. This particular project drew on several capabilities:
- Process Mapping. We leveraged the process mapping skills that we’ve developed when driving performance improvement projects over the years. We mapped the current state, then the future state, incorporating the workflow automation needed to achieve the client’s objectives around on-demand access speed.
- Data Engineering. The quality of reporting is always dependent on having clean, transformed data that is ready for analysis. In this case, the ETL pipeline needed to move and transform data without a human in the loop to ensure that the reports would be ready quickly enough, which required exhaustive testing and flawless execution.
- Workflow Automation. Using n8n, we turned the process maps into an automated workflow that was smart enough to walk participants and their Respondents through the process, addressing the many wrinkles and edge cases that would arise.
Example Process Maps
We welcome your perspective on other ways to make workplace training more effective and impactful. Drop us a line!