Introduction

When an assessment organization (ministry of education, assessment agency, state education department, etc.) decides to modernize its large-scale student assessment program, one of the initial questions to be addressed is whether to develop and maintain the digital assessment solution (i.e., the platform) in-house or to use the solution and services of an external service provider (vendor).

Over the past five years, five Canadian provinces (British Columbia, Alberta, Ontario, New Brunswick, and Newfoundland and Labrador), two territories (Yukon and Northwest Territories), and the Council of Ministers of Education, Canada [CMEC]) opted to outsource, engaging an external technology partner to modernize their large-scale student assessment programs.

The insourcing vs outsourcing decision can be challenging to address, as many factors need to be considered. This article provides a summary of advantages and disadvantages (pros and cons) of each approach and provides the reasoning behind the decisions made by assessment organizations in Canada.

Insourcing and Outsourcing^[1]

Insourcing (or in-house development) means that the assessment organization relies on its own staff, time, and resources to design, develop, and implement its own digital assessment solution. Outsourcing is a process whereby the assessment organization hires an external service provider (a vendor) to complete the work instead of using their own employees and other resources. The decision concerning whether to use insourcing or outsourcing depends on a number of variables, including the project requirements, skills needed to create and deliver the product, potential time constraints, as well as budget. The following are some of the advantages and disadvantages of each approach.

Advantages of Insourcing

• Project Control: When a project is developed in-house, the organization has complete control over all aspects of the development process. Direct access to the development team facilitates monitoring of progress, addressing issues as they arise, making process adjustments as required, and ensuring quality standards are being met. Insourcing can ensure that time and money are being allocated appropriately.

• Collaboration and Communication: Regular communication between an organization’s management and the development team, as well as continual, effective communication and collaboration among team members can contribute to the smooth, efficient operation of a project. Typically, in-house developers are in close proximity with one another (either physically and/or virtually), which can contribute to clear and open lines of communication. Furthermore, as employees of an organization, development team members are likely to have a good understanding of the project and have a vested interest in its success.

• Developing Expertise: Specialized knowledge and/or training may well be required when designing and developing a program. Taking on a project in-house can provide learning opportunities for those involved. If time and cost are not an issue, it may be in the organization’s best interest to build the team’s knowledge and expertise, which may have great value in the future.

• Potential Cost Savings: Cost is often top of mind when considering optional approaches to project development. If the organization already has personnel with the required expertise, opting for an in-house team can be less expensive than hiring an external vendor(s) to conduct the work.

Disadvantages of Insourcing

• Insufficient Talent Pool: Although an in-house development team may have a more in-depth understanding of the organization’s business, goals and requirements, there may well be a shortage, among existing staff, in the skills/competencies needed to complete the project. In the case of developing digital assessment platforms, experts are required in areas such as: knowledge of current best practices in large-scale as well as digital assessment, information technology (IT), test and data security, and psychometrics. Because of the specialized nature of many of the roles, it may not be cost-effective or practical to provide in-house staff with the specialized training needed.

• Scaling Challenges: Scaling an internal team can be challenging for organizations that need to hire personnel to staff a project or to adapt to changes in project requirements. Recruitment processes can be lengthy and may strain existing teams and resources. Difficulties in scaling development teams up or down can lead to inefficiencies and delays in the development process.

• Increased Cost: Internal staff may be suitably assigned to work on some projects; however, given the complexity and specialized nature of large-scale digital student assessments, the likelihood is that specialists will need to be hired to fill specific roles on the project. Furthermore, specialized equipment and systems may be required to accomplish the task. Given these factors, costs may be higher than anticipated, and the project may not be economically viable.

• Inefficiency: When conducting project work in-house (particularly specialized complex work such as building a digital assessment platform), there is a greater likelihood that team members will encounter issues, which will require research, training and external advice. Such problems can result in inefficiencies in the development process. Furthermore, ideally, the in-house development team will be dedicated solely to the project. However (as is often the case), if team members continue to have other responsibilities or may be called upon to assist in other areas, slowdowns may occur and a longer completion time may result.

Advantages of Outsourcing

• Superior Talent Pool: Outsourcing eliminates the need for organizations to rely on internal human resources that may lack knowledge and specialized skills in critical areas of project development. Instead, when outsourcing, organizations have the opportunity to issue Requests for Proposals (RFP) in order to select vendors that have on their rosters experienced professionals who possess the required skill sets and expertise.

• Enhanced Scalability and Flexibility: The outsourcing approach operates as an on-demand service. In this model, clients can adjust contract requirements (e.g., staffing, resources, products, timing) depending upon the changing needs of the project. Consequently, many organizations prefer outsourcing project work because of the scalability and flexibility offered by this approach.

• Cost-Effectiveness: Budget is always a major issue when launching a new project. Frequently, projects experience unanticipated problems, which could have a negative impact on the planned budget, particularly if the work is being conducted in-house. Costs associated with staffing, training and other resources involved in conducting the work may be impacted. For specialized projects, outsourcing often provides better cost-effectiveness. Relying on a contracted, external technology partner means that there will be greater predictability in the cost of the project. Hiring an external company to build a digital large-scale student assessment platform can potentially achieve the desired outcome at a lower cost than conducting the work internally.

• Enhanced Efficiency: Outsourcing can make the entire project development process operate more smoothly and efficiently. By hiring an experienced, external vendor which has the required experts with specialized skills, as well as expertise and experience in the field, there is less likelihood that serious, unexpected issues will arise. And even if they do, they can be addressed more expeditiously. Furthermore, since the vendor is external to the organization, the development team is able to focus exclusively on the project at hand, thereby reducing the likelihood the final product will not meet the client’s expectations and increasing the likelihood the project will be completed in a timely fashion.

Disadvantages of Outsourcing

• Less Control and Communication: Hiring an external service provider means that the organization must work with a new set of skilled professionals who are designated to assume important responsibilities. Although this approach allows the leadership team and in-house employees to focus on the organization’s core business, it creates challenges associated with ensuring proper oversight and quality-control procedures for the project. In addition, effective communication between the organization and service provider may be harder to maintain when outsourcing as opposed to insourcing the project. These challenges, however, can be addressed by instituting effective communication and project monitoring/oversight procedures.

Making the Decision

The decision concerning whether to choose insourcing or outsourcing to modernize large-scale assessment programs involves several factors, including the:

• assessment program’s requirements,

• project’s level of complexity,

• internal staff and associated expertise,

• project budget,

• expected timeline, and

• organization’s long-term strategy and goals.

The Canadian Experience

Rationale for Outsourcing

As was previously mentioned, over the past five years, five Canadian provinces, two territories, and the CMEC decided to modernize their large-scale assessment programs by either enhancing their already-existing digital assessments or transitioning their assessments from a paper-and-pencil to a digital format. In each case, the jurisdiction/organization has adopted an outsourcing approach by contracting with their preferred technology partner, Vretta.

To gain an understanding of the rationale for outsourcing, the author conducted an online survey with representatives of the aforementioned authorities. (Four of the eight potential respondents completed the survey.) With reference to a three-point scale (Unimportant, Somewhat Important, Very Important):

All respondents identified the following factor as being very important in making the outsourcing decision:

• Vendor Expertise (talent pool and expertise surpasses that of in-house staff)

All respondents identified the following factor as being either very important or somewhat important in making the outsourcing decision:

• Lack of Internal Resources (e.g., staff, knowledge, technology)

Three out of four respondents considered the following factors as being either very important or somewhat important in making the outsourcing decision:

• Timelines (project timelines were tight; expectation the vendor would deliver the product in a timely fashion)

• Cost-Effectiveness (outsourcing expected to cost less than developing the platform in-house)

• Enhanced Efficiency (vendor able to focus exclusively on the project; expectation that the vendor’s expertise would result in smooth, timely running of the project)

• Experience of Other Jurisdictions (positive experience of outsourcing observed/reported from other jurisdictions)

Outsourcing Types and Canadian Practice

Typically, outsourcing is of two types: “off-the-shelf” and “customized.” Generally developed by assessment technology companies to address common requirements, off-the-shelf digital assessment solutions are ready-made and can be used as is by the assessment organization (the purchaser). On the other hand, customized digital assessment solutions are designed to address the specific and unique needs of the assessment organization. They are customizations made, as per the unique requirements of the assessment organization, to an existing widely used digital assessment solution.

In the Canadian context, each assessment organization has unique requirements, resulting in the digital assessment solution requiring some form of customization. The digital assessment solution from Vretta has therefore been configured and customized to the unique requirements of each assessment organization.

Customizations are typically quite extensive, involving unique user journeys (the user’s experience as they navigate through the various stages of interaction with the digital assessment solution). Customizable user environments and processes are associated with all stages and aspects of the digital assessment cycle, including:

• item authoring and assessment delivery models;

• registration and administration processes,

• marking and scoring;

• data analytics; and

• reporting.

Following customization of the digital assessment solution, Canadian assessment organizations operate the digital assessment solution as per their requirements in partnership with their technology partner who provides development and operational guidance to support as required.

Conclusion

When a jurisdiction decides to modernize its large-scale student assessment program, an important question to be addressed is whether to conduct the development work in-house or outsource it. This article provides a summarized description of the pros and cons of each approach and explains why the CMEC, as well as the Canadian provinces and territories that have recently modernized their large-scale student assessment programs, have opted to outsource the development work. Since each assessment organization has unique requirements, a significant amount of customization has been required. Furthermore, once customizations have been implemented, they conduct the assessment functions with support from the vendor, as required. In the not-too-distant future, additional Canadian jurisdictions will move forward with modernizing their assessment programs, and it will be interesting to observe the approaches they adopt to develop and implement their projects.

About the Author

Dr. Richard Jones has extensive experience in the fields of large-scale educational assessment and program evaluation and has worked in the assessment and evaluation field for more than 35 years. Prior to founding RMJ Assessment, he held senior leadership positions with the Education Quality and Accountability Office (EQAO) in Ontario, as well as the Saskatchewan and British Columbia Ministries of Education. In these roles, he was responsible for initiatives related to student, program and curriculum evaluation; education quality indicators; school and school board improvement planning; school accreditation; and provincial, national and international testing.

Richard began his career as an educator at the elementary, secondary and post-secondary levels. Subsequently, he was a researcher and senior manager for an American-based, multinational corporation delivering consulting services in the Middle East.

Feel free to reach out to Richard “Rick” at richard.jones@rmjassessment.com (or via LinkedIn) to inquire about best practices in large-scale assessment and/or program evaluation.

References

[1] The sources of Information in this section are as follows: Duran, N. (2024). In-House VS External Software Development: Pros and Cons. Retrieved August 24, 2024 from: https://www.isucorp.ca/blog/in-house-vs-external-software-development-pros-and-cons and Indeed Editorial Team. (June 24, 2024). Benefits of In-House vs. outsourcing (With Definitions). Retrieved August 24, 2024 from: https://ca.indeed.com/career-advice/career-development/in-house-vs-outsourcing.

---