research and development model addie

Access to 13 certificate programs,
courses and all future releases

Personal Coaching and Career Guidance

Community and live events

Resource and template library

• ADDIE Model Explained: All You...

ADDIE Model Explained: All You Need to Know [+ FREE Template]

A military training framework might not be the first thing that comes to mind when designing employee learning solutions. Yet, the ADDIE model has proven its versatility and effectiveness far beyond its initial scope, becoming a cornerstone of instructional design worldwide.

What is the ADDIE model?

Purpose of the ADDIE model

• Creating a structured process for instructional design: The ADDIE model provides a clear, step-by-step framework that guides instructional designers through the process of creating educational programs, ensuring that they consider and address all critical aspects of instructional design.
• Aligning the instructional activities with learning objectives: By starting with a thorough analysis phase of the ADDIE model, you can fully align all instructional materials and activities with the learning objectives, making it more likely that these objectives will be met.
• Facilitating data-driven decision-making: By systematically collecting and analyzing data at each stage, the ADDIE model supports data-driven decision-making, allowing instructional designers to make informed adjustments that enhance the learning experience.
• Facilitating effective communication among stakeholders: By providing a common framework, the ADDIE model facilitates clear and effective communication among all stakeholders involved in the instructional design process, including educators, designers, and learners.
• Promoting continuous improvement: The evaluation phase of the ADDIE model allows for the collection of feedback and data on the training program’s effectiveness, providing opportunities for continuous improvement and keeping the training relevant and impactful over time.
• Enhancing instructional design expertise: The process of continuous evaluation and improvement helps instructional designers and educators to refine their skills and expertise over time, leading to higher-quality educational content and more effective teaching strategies.

Advantages and disadvantages of the ADDIE model

Advantages of the addie model.

• Adaptability : ADDIE instructional design model is highly adaptable and can be used across industries, disciplines, and learning environments. It can be tailored for different scale projects as well as individual or group learning.
• Consistency : ADDIE model of training provides a structured and consistent approach to instructional design, which can boost efficiency and consistency in the quality of course development.
• Iteration opportunities : ADDIE is an iterative model, which means it allows for feedback and changes at each stage of development to ensure the final product meets the desired learning objectives.
• Evaluation component : There is an evaluation component to the ADDIE model that allows businesses to measure the effectiveness of the instructional content. That is useful in identifying key areas of improvement for future iterations. 

Disadvantages of the ADDIE Model

• Linear process : The ADDIE method follows a linear process that may not be flexible or creative enough to address complex learning needs.
• Being resource-intensive : Using the ADDIE model for instructional design can be a lengthy process that also requires significant resources. It may be challenging for smaller organizations with fewer resources to implement.
• Lack of emphasis on user experience : While organizations have been increasingly focusing on digital employee experience , including in training, the ADDIE model doesn’t have a strong focus on user experience. That can lead to unengaging instruction and poor learning outcomes for employees.

The 5 phases of the ADDIE model

• What is the purpose of the training?
• Why should we do it?
• What is the desired change?
• Will the training be effective in creating this change?

Design 

“Technology is always evolving, and as a result, the tools we use to create learning content are constantly changing too. However, despite these changes, the ADDIE model has remained a timeless framework for instructional design. This is because the ADDIE process describes the fundamental steps needed to develop a learning program, which apply regardless of the tools or technology being used. While specific tools and methods may vary, the ADDIE model provides a structure for the design, development, and delivery of effective learning programs that has stood the test of time. It’s no surprise that ADDIE has become the standard for learning content production worldwide.” Nikola Velickovic, Learning Consultant at AIHR

HR Tip “Incorporate interactivity and engagement into your training materials when using the ADDIE model. This can be achieved through activities, assessments, and simulations, which help reinforce learning and create a more immersive learning experience for your learners.” Anchal Dhingra, Learning Consulting Manager at AIHR

ADDIE model examples

Training for sales representatives.

Training for public speaking & presentations

How to use ADDIE model: Best practices

• Thoroughly analyze before designing: Begin with an in-depth analysis to understand the learners’ needs, the specific problems to be addressed, and the learning environment. This foundation ensures that the training is targeted and relevant.
• Set clear, measurable objectives: Establish clear and measurable learning objectives that align with the identified needs. This clarity guides the development process and helps in evaluating the training’s effectiveness.
• Utilize an ADDIE model template for task and progress tracking: Implementing an ADDIE model template can significantly enhance project management by clearly dividing tasks among team members and tracking progress through each phase. This approach promotes clear communication, timely completion, and early identification of issues, creating a cohesive and efficient project workflow. You can download your free ADDIE model template below .
• Incorporate flexible and creative instructional design solutions: While maintaining structure, infuse creativity and flexibility into your instructional design to cater to diverse learning styles and complex learning needs. This approach can enhance engagement and accommodate various instructional challenges.
• Utilize the iterative nature of the ADDIE model: Seek feedback at each stage and make informed adjustments. This iterative process allows for continuous refinement and improvement of the training program.
• Leverage technology appropriately: Make informed decisions about technology use, selecting tools that enhance learning without overwhelming or excluding participants. Stay updated on educational technology trends to find innovative solutions that align with your training goals.
• Develop a robust implementation plan: Ensure a smooth rollout by preparing a detailed implementation plan that covers all logistical aspects, including technology setup, facilitator training, and learner support mechanisms.
• Conduct comprehensive evaluations: Beyond measuring learning outcomes, evaluate the training’s impact on job performance and organizational goals. Use these insights to inform future training initiatives and contribute to a culture of continuous learning and development.

Free ADDIE model templates

Addie model template – excel.

ADDIE model template: Powerpoint

ADDIE vs rapid instructional design

• Definition – Initial definition of learning goals and requirements
• Prototyping – Rapid prototyping of a proof of concept
• Evaluation – Evaluation of the prototype with stakeholders, followed by iterative improvements and adjustments of goals and requirements based on the POC
• Implementation   – Implementation of the adjusted goals and requirements in an upgraded version of the POC
• Repeat – Steps 2-4 are repeated until the learning goals are achieved

Before you go

Weekly update.

Stay up-to-date with the latest news, trends, and resources in HR

Erik van Vulpen

Related articles.

Payroll Audit: Checklist & How To Run It [Free Template]

Free PESTLE Analysis Templates and Actionable Guide 

14 Compensation Philosophy Examples [+ Free Template]

New articles.

360 Recruitment: Your Ultimate 2024 Guide

People and Culture vs. HR: What’s the Difference?

54 Employee Feedback Survey Questions To Ask (In 2024)

Subscribe to our weekly newsletter, are you ready for the future of hr.

Learn modern and relevant HR skills, online

InstructionalDesign.org

Home » Instructional Design Models » ADDIE Model

ADDIE Model

The ADDIE model is the generic process traditionally used by instructional designers and training developers. The five phases—Analysis, Design, Development, Implementation, and Evaluation—represent a dynamic, flexible guideline for building effective training and performance support tools. While perhaps the most common design model, there are a number of  weaknesses to the ADDIE model  which have led to a number of spin-offs or variations.

It is an Instructional Systems Design (ISD) model. Most of the current instructional design models are spin-offs or variations of the ADDIE model; other models include the Dick & Carey and Kemp ISD models. One commonly accepted improvement to this model is the use of rapid prototyping. This is the idea of receiving continual or formative feedback while instructional materials are being created. This model attempts to save time and money by catching problems while they are still easy to fix.

Instructional theories also play an important role in the design of instructional materials. Theories such as behaviorism,  constructivism , social learning and cognitivism help shape and define the outcome of instructional materials.

In the ADDIE model, each step has an outcome that feeds into the subsequent step.

Analysis > Design > Development > Implementation > Evaluation

Analysis Phase

In the analysis phase, instructional problem is clarified, the instructional goals and objectives are established and the learning environment and learner’s existing knowledge and skills are identified. Below are some of the questions that are addressed during the analysis phase:

* Who is the audience and their characteristics? * Identify the new behavioral outcome? * What types of learning constraints exist? * What are the delivery options? * What are the online pedagogical considerations? * What is the timeline for project completion?

Design Phase

The design phase deals with learning objectives, assessment instruments, exercises, content, subject matter analysis, lesson planning and media selection. The design phase should be systematic and specific. Systematic means a logical, orderly method of identifying, developing and evaluating a set of planned strategies targeted for attaining the project’s goals. Specific means each element of the instructional design plan needs to be executed with attention to details.

These are steps used for the design phase:

* Documentation of the project’s instructional, visual and technical design strategy * Apply instructional strategies according to the intended behavioral outcomes by domain (cognitive, affective, psychomotor). * Create  storyboards * Design the user interface and user experience * Prototype creation * Apply visual design (graphic design)

Development Phase

The development phase is where the developers create and assemble the content assets that were created in the design phase. Programmers work to develop and/or integrate technologies. Testers perform debugging procedures. The project is reviewed and revised according to any feedback given.

Implementation Phase

During the implementation phase, a procedure for training the facilitators and the learners is developed. The facilitators’ training should cover the course curriculum, learning outcomes, method of delivery, and testing procedures. Preparation of the learners include training them on new tools (software or hardware), student registration.

This is also the phase where the project manager ensures that the books, hands on equipment, tools, CD-ROMs and software are in place, and that the learning application or Web site is functional.

Evaluation Phase

The evaluation phase consists of two parts: formative and summative. Formative evaluation is present in each stage of the ADDIE process. Summative evaluation consists of tests designed for domain specific criterion-related referenced items and providing opportunities for feedback from the users.

Document courtesy of Wikipedia.org

Create Your Course

What is the addie training model (template + examples), share this article.

Instructional designers are responsible for all elements of building new course content. With so many components involved in the design of a successful course , designers needed a way to effectively build and measure how to do so consistently. ADDIE is a 5 step framework used in instructional design . Over the years, it has morphed from a linear approach to a more circular approach, as instructional designers have begun creating iterations of their courses. And it functions well whether your course is going to be offered online or in a physical classroom.

Skip ahead:

What are the 5 steps of the ADDIE training model?

Examples of addie model in training plans, is sam an alternative for addie, how to implement the addie training model.

ADDIE is an acronym which stands for the 5 steps in the framework, including:

• Analysis – Doing your research to plan for the course
• Design – Critical decisions will be made about the course and how it will be delivered
• Development – W here the actual course creation occurs
• Implementation – When you will begin loading content into a learning management system (LMS)
• Evaluation – Based on your findings during the evaluation phase, you’ll go back to refine your training through analysis, design, and implementation all over again

In this step, you are doing your research to plan for the course . In a traditional business or education setting, this is the step where you are reviewing existing materials, searching for knowledge gaps, and determining what factors worked well or not so well . As an individual course creator, the analysis step can be conducted through focus groups or by joining social media communities to determine the needs of your target audience. To pull some insights from your group, consider asking questions like:

• What is the course about?
• When is the course launching?
• Why is this course needed?
• How will objectives be achieved?

The analysis step is often overlooked. Taking the time to do a thorough analysis can save time and money later in the process as it guarantees your course is going to be more aligned and value-providing to your students’ learning goals . Rather than build a course you think your audience wants, use the analysis to build the course you know they need.

Design is the second step in the ADDIE training mode and it goes hand in hand with development (the next step) . In the design step, critical decisions will be made about the course and how it will be delivered. Some questions to ask yourself or your learning group include:

• What learning barriers do students face?
• Will the course be video only or will there be interactive components?
• How can the course be made accessible for different learning needs?
• Will the course be a blended mix, with some content delivered live and other elements pre-recorded?
• Will there be cohorts?
• What learning sequence will be used?

Different delivery strategies will impact the overall course design and potentially your budget. The more complex course features you add (such as interactive quizzes or custom certificates), the more expensive your course development will become. 

Read more: How to Plan an Online Course (+Templates)

After determining how the course will be delivered, the next part of design is to determine the order that course content will be delivered. This is a great time to put together a small focus group and gather feedback about the design. One example of course order delivery would be that content topics build on each other from more introductory and high-level topics to more advanced or niche topics. Alternatively, courses could start with the most complex topics first and break down the subject in following segments.

Once you are confident that you have the elements you need, it’s time to create a storyboard. In simplest terms, the storyboard is the roadmap for your course and can be used to keep everyone working on the course organized and working towards the same goal. If you have never used a storyboard, here’s a great introductory article about how to create an eLearning storyboard (including templates!) 

Development

The development stage is where the actual course creation occurs. There is no “right” way to do this. One thing to keep in mind, if you gather all your assets up front, the development portion of creating your course will go more quickly. 

In the development step, be prepared to test and review frequently. Check for accuracy of content, the look, feel, flow of the course, and then you will be ready to implement. Again, you might want to get a pulse check from your learning group to make sure that what you’ve developed is aligned with their expectations. Feel free to spend time here testing different layouts and visual elements to make sure that the course content is easy to digest. However, don’t get too wound up in the nitty details. Because in all honesty, who hasn’t lost a few hours searching for the perfect image?

Implementation

The implementation stage is when you will begin loading content into a learning management system (LMS) such as Thinkific . During implementation, you will also be checking to ensure all content functions work properly.

Some things to specifically consider include:

• Students can find the course
• Students can enrol
• Course content is easy to access after enrollment
• Integrations work properly
• Any livestream or notification reminders go out without fault
• Instructors can view registration and engagement analytics
• Certificates or reports post-course are downloadable and custom to each student
• There is a clear, identified way for students to ask questions to the instructor (and the instructor can respond)

While listed as the fifth and final step of the ADDIE training model , the evaluation will run iteratively as long as your course is live. This is important to ensure features are always working, the course content aligns with students’ expectations and learning goals, and instructors are able to efficiently engage with students. 

During the evaluation, focus on whether the course has met the goals for the course, implementing feedback from the learners, and potentially making content changes or updates. Based on your findings during the evaluation phase, you’ll go back to refine your training through analysis, design, and implementation all over again! This process should be repeated at least every two months, or whenever you notice changes need to be made to course content (if industry best practices or school curriculum requirements shift, for example). 

It’s one thing to talk about ADDIE and another to see what ADDIE looks like in action. Here are two examples of how to build an ADDIE training model to be used by companies and entrepreneurs.

ADDIE Training Plan for a corporate training audience

• Course goal
• Inventory existing content
• Work with business partners to determine outcomes
• How will course be delivered? In-person, online, or hybrid?
• Who will be delivering the content?
• What is timeline for creation?
• What tools are being used to create?
• Storyboard and collect assets
• Gather assets
• Record video
• Record audio
• Create in authoring tool
• Create worksheets, etc.
• Follow storyboard
• Online or hybrid load to LMS
•  In-person schedule sessions
• Assign Learners
• Track completion
• Survey learners
• Evaluate if goals are met
• Evaluate behavior change
• Determine changes needed and revise course

Addie Training plan example for entrepreneur audience

• Conduct focus groups/poll existing groups
• Research similar courses online
• Determine course goals/needs
• Determine content delivery method
• Will there be interactive elements or straight video?
• What type of activities will be included?
• Begin storyboarding/collect assets
• Upload content to LMS
• Open course for enrollment
• Evaluate if goals were met
• Evaluate sales
• Evaluate repeat business

SAM stands for “ successive approximations model ,” and is like a “close cousin” instructional design approach to ADDIE. While similar to ADDIE, SAM is based on an agile development cycle with ongoing feedback and iterations of a content rather than the extended cycle found in ADDIE. You could opt to use SAM instead of ADDIE if you are comfortable with multiple steps happening at the same time and committing to rapid collaboration efforts with your client. There is not a simple answer as to which one is better. It completely depends on the process you prefer and the type of instructional design program that you are working within. 

Make use of existing processes

If you review your existing course design processes, you may find that you are using a less structured version of ADDIE already. It is important to recognize this, as you may be able to make use of some existing procedures that you are already familiar with. 

Avoid skipping steps

A challenge with the ADDIE model is that many organizations focus t oo strongly on just the design, development, and implementation phases and often skip the analysis and evaluation phases. This happens because business requirements ask for courses to be created quickly to start generating additional revenue streams for the company. In doing so, they ignore time-consuming steps that actually improve the quality of the course in the long-run. And it happens all too often.

Involve stakeholders early

To avoid the pitfall mentioned above, talk to your stakeholders and schedule regular conversations regarding training needs. When these conversations occur, the stakeholders may discover that training isn’t always the answer. When training is the answer, you can work together to create meaningful plans that are based on the values of the students you want registering for your courses.

Having these conversations can be challenging, and I suggest easing into them, asking clarifying questions about the goal of the training, and providing solutions for how to achieve the goal. Regular check-in meetings also help ease the conversation and provide a set time for checking-in on the priority and status of this initiative.

Do your research

When possible, do some research beforehand. This could be as simple as looking at similar courses already in your LMS, gathering usage data, or talking to managers in operations to find out what their needs are. Even spending a short period of time conducting a needs analysis will pay dividends in the end.

Read more: How to Design Your Online Course (Visually & Structually)

Get feedback often

In addition to the usual end-of-course feedback, sometimes referred to as “smile sheet” feedback, consider implementing a system for gathering targeted feedback about the effectiveness of the course and behavior changes. Make it a best practice to do this at the 30, 60, and 90 day marks. This could be in the form of a survey sent to those who have completed the course or targeted to managers to ask questions about employee retention, customer feedback, or effectiveness in the role.

Use the feedback to iterate frequently

Use feedback data to continually improve and enhance the course. It may help to set a regular cadence to integrating feedback into your course design. Depending on how fast your course content changes, you might want to consider doing this monthly, quarterly, or annually. For example, accounting principles hardly ever change. But new software development techniques are always popping up overnight, so new content will help you stay ahead of the curve.

In summary, ADDIE is a process that can be implemented to allow for better course design. The process begins by conducting a needs analysis, followed by designing the course, using tools such as a storyboard or Kanban board, then creating the content, implementing the content by uploading to the LMS, and last, but not least, evaluating the effectiveness of the course and making improvements, thus starting ADDIE again.

Ready to start your online course? Try Thinkific free today! 

This blog was originally published in December 2022 and was updated in August 2023 with more clear breakdowns, and additional resources.

Mary Nunaley co-founded the Lavender Dragon Team with her son Amadeus. Mary is an ATD Master Instructional Designer, gamification aficionado, award winning course creator, and an advocate for putting the fun back into learning.

• 10 Best Online Quiz Makers (And What Features To Look For)
• What Is Microlearning? The Case For Shorter, Bite Sized Learning
• 10 Instructional Design Models For Creating A Winning Online Course
• 25+ Best ChatGPT Prompts for Instructional Designers
• 7 Instructional Design Storyboard Templates For Online Course Creators

Related Articles

How to build a successful coaching membership site.

How do you build a successful coaching membership site? Learn strategies for creating an engaging membership site, beyond an online course

The Best Online Teaching Platforms to Earn Money in 2024

How to choose an online teaching platform for your course? Start with these free online teaching platforms and start earning money today.

Use This Free Webinar Outline Template For More Impact

With the right webinar outline template, you can plan your webinar to move your audience seamlessly towards making a purchase.

Try Thinkific for yourself!

Accomplish your course creation and student success goals faster with thinkific..

Download this guide and start building your online program!

It is on its way to your inbox

Instructional Design: The ADDIE Approach

• © 2009
• Robert Maribe Branch 0

, Dept. Educational Psychology & Instructi, University of Georgia, Athens, USA

You can also search for this author in PubMed   Google Scholar

• Utilizes a simple, yet robust organizing framework
• Uses a thematic approach to the content
• Presents the concept, theory and practice for ADDIE
• Contains a glossary
• Includes supplementary material: sn.pub/extras

85k Accesses

388 Citations

2 Altmetric

This is a preview of subscription content, log in via an institution to check access.

Access this book

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as EPUB and PDF
• Read on any device
• Instant download
• Own it forever
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Other ways to access

Licence this eBook for your library

Institutional subscriptions

About this book

Similar content being viewed by others.

Instructional Design Models

Instructional Design Methods and Practice

An Examination of the Systemic Reach of Instructional Design Models: a Systematic Review

• Instructional
• Instructional Design
• learning and instruction

Table of contents (7 chapters)

Front matter.

Robert Maribe Branch

Back Matter

Authors and affiliations, , dept. educational psychology & instructi, university of georgia, athens, usa, about the author, bibliographic information.

Book Title : Instructional Design: The ADDIE Approach

Authors : Robert Maribe Branch

DOI : https://doi.org/10.1007/978-0-387-09506-6

Publisher : Springer New York, NY

eBook Packages : Humanities, Social Sciences and Law , Education (R0)

Copyright Information : Springer-Verlag US 2009

Hardcover ISBN : 978-0-387-09505-9 Published: 05 October 2009

Softcover ISBN : 978-1-4899-8423-4 Published: 23 August 2014

eBook ISBN : 978-0-387-09506-6 Published: 23 September 2009

Edition Number : 1

Number of Pages : X, 203

Topics : Educational Technology , Learning & Instruction , Business and Management, general , Education, general

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Ready to get started?

What is ADDIE? Your Complete Guide to the ADDIE Model

In the landscape of corporate training and development, the quest for effective, impactful, and engaging training programs remains at the forefront of business leaders’ and corporate training professionals’ objectives. The ADDIE model, with its structured yet adaptable framework, offers a revolutionary approach to meeting these goals. This comprehensive guide delves into the ADDIE model, illustrating its practicality, innovation, and effectiveness in revolutionizing corporate training.

• What is the ADDIE training model?

ADDIE is a learning development model that stands for Analysis, Design, Development, Implementation, and Evaluation. It’s a more time-consuming process than the SAM model, but it’s a complete cycle focusing on getting it right the first time. With ADDIE, a solution spends more time developing, where designers tweak and perfect the details before launch.

The ADDIE model is a systematic instructional design framework used to guide the process of creating education and training programs. Standing for Analysis, Design, Development, Implementation, and Evaluation, ADDIE serves as a comprehensive roadmap for instructional designers and training developers. This iterative process begins with analyzing learners’ needs, followed by designing the instructional approach, developing the content, implementing the solution, and evaluating its effectiveness. By structuring the creation of learning experiences in this way, the ADDIE model ensures that training programs are efficient, effective, and aligned with learners’ and organizational goals.

The key phases of the ADDIE model

The analysis phase lays the foundation for successful training by identifying the learners’ needs and objectives, as well as the training’s context. In the corporate realm, this involves understanding the technical landscape, learner demographics, and the specific challenges to be addressed through training.

Designing effective training programs is at the heart of the ADDIE model. This phase focuses on creating a blueprint for the training, outlining the learning objectives, content structure, and the instructional strategies best suited for the target audience. The design phase ensures that the training will be engaging, relevant, and impactful.

Development is where the learning experience design comes to life. This phase involves the creation of the training materials, including multimedia content, assessments, and learning activities. It emphasizes the importance of tailoring the content to meet the learners’ preferences and the learning objectives established in the design phase.

Implementation is the execution phase, where the training program is rolled out to the learners. This stage tests the practicality and effectiveness of the training design and development, highlighting the importance of a smooth delivery mechanism and the readiness of both instructors and learners.

Evaluation is critical for assessing the effectiveness of the training program and identifying areas for improvement. This phase involves gathering feedback, analyzing learning outcomes, and determining the training’s impact on performance. It ensures that the training not only meets the current needs but is also refined for future iterations.

The pros and cons of the ADDIE model

The ADDIE model offers a comprehensive framework for instructional design , with strengths in its structured approach and flexibility. However, it’s important to acknowledge both its advantages and limitations. While ADDIE provides a systematic methodology for creating effective training programs, its time-consuming nature and the potential for rigidity in its sequential phases are points of consideration.

• Pros of the ADDIE model:
• Structured framework: ADDIE offers a clear, systematic approach that guides instructional designers through each phase of training development.
• Flexibility: While structured, it allows for adaptability to different learning environments and needs.
• Comprehensive analysis: It emphasizes the importance of understanding learner needs, environmental factors, and objectives at the outset, leading to more targeted and effective training solutions.
• Iterative process: It encourages continuous evaluation and refinement, enhancing the quality and effectiveness of training programs over time.
• Widely recognized and used: As a well-established model, it is familiar to many instructional designers, facilitating collaboration and communication among professionals.
• Cons of the ADDIE model:
• Time-consuming: The detailed and sequential nature of the model can lead to longer development times compared to more agile methodologies.
• Potential for rigidity: Its linear progression through phases can sometimes limit creativity and rapid response to changing needs unless intentionally managed for flexibility.
• Resource intensive: Comprehensive analysis, development, and evaluation phases may require significant resources in terms of time, personnel, and costs.
• Delayed testing: Since testing occurs later in the process, there may be less opportunity for early identification of issues or for incorporating feedback without revisiting and revising several stages.
• Assumes static needs: The model is based on the assumption that training needs and objectives remain constant throughout the development process, which may not align with the dynamic nature of some organizations and learning environments.

By considering these pros and cons, organizations and instructional designers can better decide when and how to apply the ADDIE model to meet their training development needs effectively.

• ADDIE in action: Applications in corporate training

The application of the ADDIE model in corporate training has seen numerous successes. It facilitates the creation of tailored training solutions that address specific organizational challenges, enhance employee skills, and foster a culture of continuous learning and development. Incorporating the ADDIE model into corporate training can take many forms, depending on the specific needs and context of an organization. 

Below are practical examples of how the ADDIE model can be successfully applied across various industries to enhance training programs and outcomes.

• Analyze: A tech company identified a gap in its sales team’s ability to sell a new product line effectively. The analysis phase involved surveys and interviews with sales representatives to understand their challenges and learning needs.
• Design: Based on the analysis, the training program was designed to focus on product knowledge, sales techniques, and customer engagement strategies. Interactive eLearning modules and role-play exercises were selected as the primary training methods.
• Develop: The training materials developed included video tutorials, a product knowledge database, and simulation exercises to provide hands-on experience with selling scenarios.
• Implement: The training was rolled out through the company’s learning management system (LMS), with sales representatives required to complete modules at their own pace and participate in live virtual role-playing sessions.
• Evaluate: Post-training, sales performance was monitored through sales metrics, and feedback was collected from participants. The evaluation showed improved sales outcomes and product knowledge, leading to the training being adapted for ongoing use with new hires.
• Analyze: A financial services firm aimed to enhance its leadership pipeline by developing a program for high-potential employees. The firm conducted a needs assessment to identify key leadership competencies required for its future growth.
• Design: The leadership development program was designed to include workshops, mentoring, and project-based learning activities focused on strategic thinking, decision-making, and team leadership.
• Develop: Development efforts resulted in a comprehensive curriculum, incorporating expert-led workshops, case studies, and a mentoring program pairing participants with senior leaders.
• Implement: The program was implemented over a six-month period, with participants engaging in various learning activities and applying their skills in leadership projects.
• Evaluate: The effectiveness of the program was evaluated through participant feedback, assessments of leadership competencies before and after the program, and the impact on participants’ career progression. The positive outcomes led to the program becoming a cornerstone of the firm’s talent development strategy.
• Analyze: A healthcare organization needed to ensure all staff were up-to-date on new compliance regulations. The analysis phase involved reviewing regulatory requirements and assessing current staff knowledge levels.
• Design: The compliance training program was designed to be accessible and engaging, utilizing scenarios and quizzes to highlight key compliance issues. The program was segmented into modules specific to different staff roles.
• Develop: Development included creating interactive online training modules with real-life scenarios healthcare staff might encounter, focusing on practical application of compliance rules.
• Implement: The training was implemented as mandatory for all staff, with progress tracking through the organization’s LMS. Reminders and support were provided to ensure high completion rates.
• Evaluate: Post-implementation, the organization evaluated the training’s effectiveness through knowledge assessments and compliance audits. Feedback led to adjustments in the training content and approach to further enhance comprehension and application of compliance practices.

These examples illustrate the versatility and effectiveness of the ADDIE model in addressing diverse training needs across various sectors, demonstrating its value in creating tailored, impactful corporate training programs.

While specific project details and the internal processes of companies can be proprietary or confidential, many organizations across various industries have publicly acknowledged or demonstrated principles aligned with the ADDIE model in their Learning & Development (L&D) and training strategies. Below are examples of sectors and types of companies known to implement ADDIE or similar systematic instructional design models for their training programs:

Technology and software companies

• IBM: Known for their robust L&D programs, IBM utilizes structured approaches to instructional design that closely resemble the ADDIE model for both their internal training programs and its customer education initiatives.
• Microsoft: Microsoft implements systematic training development processes for both software training and professional development of their employees, focusing on continuously analyzing and evaluating the effectiveness of their training programs.

Financial services

• Bank of America: They have been recognized for commitment to employee development and training, using structured models to design, develop, and deliver training programs that enhance skills and performance.
• JPMorgan Chase: They utilize a systematic approach to develop and implement training that supports both regulatory compliance and professional growth, indicating a methodology similar to ADDIE.
• Kaiser Permanente: Their focus on comprehensive needs analysis and evaluation in their training programs ensures healthcare professionals receive effective and current training, a hallmark of the ADDIE model.
• Johnson & Johnson: Employment of structured training designs provides employees with continuous learning opportunities, focusing on innovative and effective healthcare solutions and professional development.

Education and eLearning

• Coursera & edX: While not traditional “companies” in the sense of product manufacturing, these platforms work with universities and corporations to develop online courses, often employing systematic instructional design models to ensure course effectiveness and learner satisfaction.
• Khan Academy: Known for its educational content across a wide range of subjects, Khan Academy likely utilizes phases of the ADDIE model to design, develop, and evaluate its instructional materials for maximum impact.

Manufacturing and retail

• Toyota: Toyota incorporates structured training systems to uphold high standards of quality and efficiency, emphasizing continuous improvement (Kaizen), which aligns with the evaluation and analysis stages of ADDIE.
• Walmart: They have implemented sophisticated training and development programs, including virtual reality-based training, that are likely developed using a systematic approach to instructional design to meet the diverse needs of its workforce.

These examples demonstrate the widespread adoption and adaptability of the ADDIE model across different sectors. Companies appreciate the model’s structured yet flexible framework for creating, implementing, and refining effective training and development programs to meet specific organizational needs and goals.

• Comparing ADDIE with other training models

When juxtaposed with other training models, such as other iterative design models like the SAM (Successive Approximation Model), ADDIE stands out for its comprehensive and methodical approach. While SAM offers a more agile development process, ADDIE’s strength lies in its thoroughness and emphasis on analysis and evaluation, making it particularly suited for complex training needs.

This comparison chart highlights the core differences between ADDIE and other models in instructional design. The choice between models largely depends on the specific needs of the project, including the scope, timeline, resources, and flexibility required. For example, ADDIE is well-suited for projects with clear objectives and stable requirements, while SAM excels in environments where rapid development and adaptability are key.

• Final thoughts on ADDIE

The ADDIE model is a powerful tool in the arsenal of corporate training professionals and business leaders. Its structured approach to instructional design not only addresses the pain points of developing engaging and impactful training programs but also aligns with the core motivations of implementing effective, research-backed training methodologies. By embracing the ADDIE model, organizations can enhance their training initiatives, ultimately leading to improved employee engagement and knowledge retention.

Discover how ELM Learning can help you apply the principles of the ADDIE model to create customized, effective training solutions for your team. Explore our custom eLearning solutions.

Sign up to get the latest industry tips, insights, and trends from our team of learning experts.

Share this article:

The ELM L&D Hub

Instructional design knowledge for all.

ADDIE model

The ADDIE model is a well-known and widely adopted instructional design tool for increasing the success of new training programs.

Julia Kuzmina

Content Marketing Specialist, Valamis

June 22, 2022 · updated July 10, 2024

13 minute read

Organizations need to continually develop their employees with additive training programs. Technology and customer expectations are constantly evolving. Therefore, to compete in today’s marketplace, you require a talented and up-to-date workforce.

This places a significant burden on instructional design and producing learning and development courses with tangible results. The ADDIE model is a well-known and widely adopted tool that can increase the success of new training programs.

In this blog, we will take you through:

• What is ADDIE training model?

Why ADDIE model?

A brief history of the addie model.

• The five steps of the ADDIE model
• ADDIE model example – Corporate training

Advantages and disadvantages of ADDIE

• ADDIE vs. SAM – Which model best suits your learning and development project?

What is the ADDIE training model?

ADDIE is a popular model for instructional design, providing a framework to produce effective learning and development programs. “ADDIE” itself is an acronym for the five stages of the model:

Development

Implementation.

These five clearly defined processes help ensure training programs and their resources have the required elements to succeed.

In addition, they give instructional designers a structure to rely on and consider as they progress through the development of new learning and development projects.

Research shows that ADDIE methodologies offer a systems-based, iterative learning and development design strategy while incorporating feedback and review to improve.

When designing a new training program, it can be easy to get lost in the details.

The ADDIE model provides a sequential list of requirements with known inputs and outputs such that staff developing the course can maintain a big-picture view of what they are trying to achieve.

Many organizations already implement aspects of the ADDIE model without it becoming a formalized design structure. However, due to the fast-paced nature of business, it is common for teams to focus on the design, development, and implementation stages.

Unfortunately, this approach often fails to capture the original need of the program and the impact it has on employees. While perhaps requiring additional time and examination, ADDIE goes further to understand the goals and outcomes of new learning and development courses.

The ADDIE model is generic and applicable to any type of learning experience, audience, or industry. The fact that it is not industry-specific has helped ADDIE become one of the most common design models in use. Several other models for instructional design are based on variations of the ADDIE model.

The Centre for Education Technology at Florida State University first introduced the ADDIE model in 1975 for the US army. Its use later spread across all other branches of the US armed forces.

ADDIE is based on a previous instructional design model from the US Air Force known as the “Five-Step Approach.” While the ADDIE model holds on to the five steps, it breaks down each of its broad processes into many more substages for greater detail.

ADDIE has evolved over the years, with revisions to the five stages, making the model more dynamic and interactive. However, a version of ADDIE similar to what we know today has been in use since the middle of the 1980s.

The 5 steps of the ADDIE model

Step 1. analyze.

The first step of the ADDIE model is setting goals for the new program and researching the intended target audience. This includes the audience’s existing knowledge and skills, future training needs, and the appropriate training environment and methods that organizations could deploy.

Learning and development courses must have a clear goal matching the audience’s skills and intelligence to succeed.

By taking the time to analyze the course’s recipients, you can ensure pre-existing knowledge is not being re-delivered and that the course outcome matches the business’s real-life needs.

The analysis step of the ADDIE model typically requires researching the organization’s existing learning and development material, identifying potential problems or knowledge gaps, and reviewing previous programs to see what was successful in the past. This means effective communication with both management and their employees (potentially through surveys or focus groups) to understand the existing situation.

A simple way of identifying areas of improvement is gap analysis – comparing the desired situation to your current situation. This could be related to sales and financial performance, becoming more efficient and embracing new technologies (digital transformation), improving company culture through new diversity initiatives, or something completely different.

To manage, plan, and monitor existing and desired skills for a role, team, department, project, or an entire company you can use the Skill matrix framework .

Suppose the root cause of the problem is employees lacking specific knowledge, skills, or the right mindset. Organizations could make tangible improvements with new learning and development programs in that case. Skill Matrix template is a good start to identify missing competencies and find potential skill gaps in your organization.

Skills matrix template

Efficiently assess, manage, and maximize your team’s potential and streamline your workflow.

Training needs analysis is another critical tool that can help organizations determine the information to deliver in the new program and define the goals that will feed into every future step of the ADDIE model.

Other essential tasks an organization needs to consider during step 1 of the ADDIE model include:

• Identifying all involved stakeholders
• Understanding the future resources required for the program
• Gathering detailed information on specific audience personas for future training design

The analysis step is one often overlooked during instructional design. However, taking the time to set the foundations for your new training program gives you the base needed for the steps to come.

You should always clearly understand the problem before designing the solution.

Detailed analysis saves time and money later in the process by improving the training program’s impact.

Step 2. Design

This is where all the information gathered during the analysis step is dissected to make informed decisions about the design of the development program.

The design step requires a systematic approach to the specific learning objectives identified, the course’s subject matter and content outlines, and how it will be delivered in terms of content, exercises, media, etc.

With a systematic strategy, you can logically assess the possible scenarios in an orderly fashion to understand the best approach that achieves desired learning outcomes .

Organizations should create a high-level outline of the entire program to structure the learning interventions and specify objectives for each aspect participants receive.

It is also vital to determine how participants will be evaluated. While not every training program requires individual assessments, you should have a strategy in place to measure its impact and track its value.

Finally, before developing the program’s specifics, the design stage should conclude with buy-in from all stakeholders. This means briefings on learning objectives, their impact, and the decisions that went into the course’s outline.

This is the last step where significant changes to the program as a whole can be made, so it is imperative that stakeholders are happy with the chosen goals and processes needed to achieve them.

Step 3. Development

The development stage takes the outline defined in step 2, creates the assets required to bring it to life, and tests various methodologies for delivering the content.

While the previous two steps involve examination, planning, and generating ideas, development is when these ideas are first put into action.

It is called “development” for a reason, though, and the outcomes of this step will evolve during the process. Therefore, you should be prepared to try multiple approaches to determine the content and delivery that best fits the target audience. Also, always double-check the content’s accuracy and look at the course as a whole to ensure it flows naturally.

With the design program outline for guidance, you need to create all the required assets. This could be presentations, videos, graphics, instruction manuals, assessments, or anything else you’ll need.

Remember, there isn’t a “correct” way of getting this done, and you should consider the best option for your situation. For example, do you have the facilities to produce everything in-house, are you going to outsource the production to third parties, or a mix of both?

Step 4. Implementation

Step 4 is the actual delivery of the learning and development program.

A significant part of implementing the course is project management and getting into the details of everything the program requires, from communication, logistics, and data collection to identifying and training staff who can deliver the content in an engaging way.

Good project managers can adapt on the go to ensure they deliver the best possible product to the course participants. This could mean tweaking content for a change in audience persona or the minutiae of having everything ready to go on the day.

The training of course leaders needs to cover:

• The curriculum
• The required outcomes
• How it should be delivered
• The use of course assets and resources
• Any assessment/feedback procedures

However you choose to assess the impact of the training program, the implementation phase is your chance to gather as much data as possible for evaluation.

Step 5. Evaluation

Evaluation, the final stage of the ADDIE model, is when you determine the performance of the training program. You want to go into detail on every aspect of the project:

• What did the participants learn?
• Can they apply these skills in their daily work?
• Were participants engaged and motivated to learn?
• Did the program meet its goals?

The primary objectives of the evaluation step are to determine if the initial goals set way back in step 1 were met and what could be improved.

The findings from this step feed back into the next project’s analysis stage, allowing you to refine training practices and enhance your success rate and efficiency moving forward.

Evaluation can be separated into two parts: formative and summative . Formative evaluation happens during every step of the ADDIE model to keep the program on track. Summative evaluations refer to specific testing assessed at the end to evaluate what participants learned and the program’s effectiveness. Staff running the course should send the evaluation results to all relevant stakeholders.

An important consideration at the end of a new learning and development program is assessing the business case for future work. Does the program generate enough value to warrant further funding?

Furthermore, L&D professionals increasingly use learning analytics and automation to identify the weak points of learning, patterns and potential of employees.

Training evaluation form

Get a handy printable form for evaluating training and course experiences.

ADDIE model example – Corporate Training

The ADDIE model is a general-purpose approach to instructional design. However, to get the ideas and methodology across, it can help to be more specific and look at an example.

Below is a demonstration of the specific outcomes from the five ADDIE steps in the case of corporate training – the process of training employees via various techniques or activities.

• Gap analysis
• Problem identification
• Catalog of existing material related to the subject matter
• An understanding of the target audience and their pre-existing knowledge/skillsets
• List of all relevant stakeholders
• Resources that needed for the program
• The big picture goals of the learning and development program
• Defined learning objectives
• High-level course outline
• Delivery method (interface and environment)
• Course progression
• Timeframe for each activity
• Tools required to create course assets
• Assessment/feedback strategy
• Alignment with relevant stakeholders
• Creation of necessary course assets
• Test and evaluate different assets and delivery approaches
• Finalize and quality check course details
• Train staff to deliver the program
• Project management (logistics, scheduling, communication, resources, etc.)
• Track the program and its performance in real-time
• Data collection
• Analyze collected data
• Assess the program’s performance (content delivered, behavioral changes, etc.)
• Identify future changes to improve success/efficiency
• Structure – ADDIE provides a structured framework to guide learning and development staff through the process of instructional design.
• Provides a starting point – Often, finding a starting point is the hardest part of a project. With ADDIE, the steps and their outcomes are laid out for you.
• Goal-centric view – The ADDIE model begins with detailed research to define the program’s overall goals. This ensures objectives remain front and center throughout the following steps.
• Industry/audience/goal agnostic – ADDIE is a versatile model applicable to any industry or audience and not dependent on a specific goal.
• Self-improving – With detailed evaluation feeding into future analysis, the ADDIE model ensures learning and development programs improve over time.

Disadvantages

• Time-consuming and costly – The level of detail required before implementing the program means the ADDIE model can be a time-consuming and expensive process.
• Linear and inflexible – With a rigid linear structure, it can be hard to adapt to changes and react to new information.
• Audience accuracy – Unlike other more iterative models that quickly implement and discover learner needs during the project, the ADDIE model puts understanding the target audience at the start of the process. This can lead to difficulties trying to accurately discover the pre-existing knowledge base and the right level at which to pitch the course’s content.

ADDIE vs. SAM – Which model best suits you?

While ADDIE is a popular approach to developing learning and development projects, it is one of many models available.

SAM, stands for Successive Approximation Model, is another popular instructional design model seen as an alternative to ADDIE. The SAM has multiple variations, but they all try to introduce iteration into the design process. This includes rapid prototyping and adding development and implementation earlier in the process.

SAM attempts to counter one of the main drawbacks of the ADDIE model, the time-consuming nature of getting the program created and implemented. Instead, SAM is a rapid development model, quickly producing new training programs with periodic reviews and evaluations to iterate the process and improve course outcomes.

Comparison ADDIE and SAM models

ADDIE is a slower pace more systematic approach to instructional design compared to SAM which enables a faster path to implementation. However, early SAM programs are less polished and generally require multiple iterations and rework to achieve the best results.

The choice between the two doesn’t have to be mutually exclusive. Depending on the situation and project’s time pressures, it is helpful for organizations to have an understanding of both models. This allows them to pick the optimal approach for each given program.

The choice often comes down to the team putting together the learning and development course and how they like to work. Some prefer taking the time to fully understand the problem, then utilizing ADDIE’s more comprehensive framework to deliver a great product. Others enjoy learning by doing and seeing what fast prototype courses look like in practice before refining based on feedback.

You might be interested in

Key areas of employee development

Learning and development fundamentals

The meaning of Learning Management System (LMS)

Celebrate the spirit of independence, click now for exclusive CHRMP offers!

ADDIE Model: A Comprehensive Guide to the 5- step Instructional Design Model

Author by : chrmp.

• ADDIE model

Rapid technological advancement has transformed the designing of learning programs in the corporate world today. 

The use of well-established and effective learning evaluation models, like the ADDIE model, has made it even easier to develop effective learning programs that ensure successful training sessions that facilitate employee development. 

There are five factors of the ADDIE model. Each factor focuses on a different aspect of planning, producing, implementing, evaluating, and improving your training programs. This blog will teach you about the ADDIE model, its uses, application, and much more. 

Let’s get started.

What is the ADDIE model?

The ADDIE model is a five-step learning evaluation model used by instructional designers and trainers to design, develop, implement, and evaluate learning programs. The acronym ‘ADDIE’ stands for Analysis, Design, Development (or creating), Implementation, and Evaluation.

The ADDIE model was initially designed by the Florida State University for the U.S. Army in 1975 and has since emerged as the default process of instructional design for organisations worldwide in present times.

The ADDIE model helps develop a well-structured framework for every piece of training material created, which ensures maximum effectiveness, and enables a well-planned, organised and effectively deliverable training program. 

Besides corporate training initiatives, the ADDIE model can be applied to several types of learning material like online or offline courses, lectures, coaching sessions, information brochures, etc.

The 5-Step ADDIE Model Explained

As mentioned previously, ADDIE stands for Analysis, Design, Development, Implementation, and Evaluation. Each letter in the acronym stands for a phase of the ADDIE learning evaluation model. 

The ADDIE model needs to be completed in the sequential order in which it is present, starting from analysis to evaluation. The model is designed as a continuous process of improvement and development. 

The ADDIE model helps organise and streamline the production of course materials for training programs. One of the reasons the ADDIE model has been so successful is that it is associated with a decent quality design, crisp learning objectives and specific, structured content to get the desired learning outcomes.

We shall now look at each phase of the ADDIE model and go through the processes involved.

1. Analysis

The analysis stage focuses on all the foundational factors involved in developing a course, like a problem, training needs, the target audience and learning goals. 

In this phase, the instructors look to identify the problem to analyse training needs. There is no standardised framework for what needs to be included in the analysis phase because it varies depending on the company’s needs. 

But some primary outputs include identifying the problem and acknowledging skill gaps . A company stakeholder approaches the instructional designer with a lingering problem like diminishing sales or a lack of vital employee skills. 

The problem will be analysed to determine whether training can solve it. Eventually, other factors, such as skill gaps, current knowledge, hindrances , etc., will be evaluated to begin designing a training program. 

Some of the questions addressed in the Analysis phase include:

• What is the desired outcome of this training?
• What tools will prove best in the training program?
• How much knowledge does the target audience already have?
• How much time is needed to complete the training?

Once you have the answer to the questions like the ones mentioned above, you will have a basic framework of the training program that fits all your company’s needs. 

The main objective of the design phase is to create a well-designed curriculum. You must ensure that you have a clear idea of your company’s training needs, what the employees know, and how much they already understand. Then, you need to develop a plan to carry out the training program. 

The design phase is the most critical phase of creating an effective training program and is also the most time-intensive. It also requires excellent attention to detail.

The design phase focuses mainly on learning objectives, subject matter analysis, lesson planning, assessment, and media selection. 

According to the ADDIE model, building a storyboard or a blueprint of the entire course proves beneficial in helping instructional designers and stakeholders to visualise the bigger picture with ease while speeding up the development process. 

By the end of the design phase, a comprehensive outline of the course, along with the overall design and the storyboard/blueprint, is ready.

3. Development

With the help of the output from the analysis and design phase, the course enters the creation process in the development phase. This phase requires instructional designers to plan and test their ideas. The main goal of the development phase is to ensure that the program meets the requirements set out by the stakeholders. 

The development phase involves laying out the content in a visual format, creating graphics, choosing fonts and colour schemes, making videos and infographics, etc.

A significant part of the development phase is testing the course, which mainly involves establishing testing and review methods with the stakeholders. Testing is a critical part of the entire process; hence it’s good to have extra sets of eyes following the whole process so that no error goes unnoticed. 

By the end of the development phase, the course and initial beta testing are completed. 

4. Implementation

Once the course is completed and initial testing is done, the next stage is implementing the course. This stage is an essential part of the process. 

Some key features of the implementation phase include delivery, communication and feedback. 

Delivery of the course involves distributing it to the employees in the chosen format along with accompanying materials like guides or manuals.

Once the course is delivered and employees start using it, you will need to create a communication plan to address any issues the employees might face and receive feedback on the course. 

The instructional designers then use this information to revise the course and make necessary changes to enhance the learning experience.

5. Evaluation

The evaluation phase is the final phase in the ADDIE model. It involves measuring the efficiency and overall effectiveness of the course by collecting information to check whether the course requires more modifications to better suit learning needs. 

Collecting data in the evaluation phase concerning the effectiveness of the training program can be done through several methods like surveys, interviews, questionnaires, or focus groups.

Evaluation is a vital component of the ADDIE process because it helps answer questions like: Is the course working? Is the training program successful? The outputs are then compared with the initially set goals to evaluate the program’s effectiveness. 

Based on the analysis of the results, further changes are made to the course, or in some cases, the entire ADDIE process is implemented once more to create a new course with the required improvements.

ADDIE vs SAM: Which One is Better?

Even though the ADDIE model is widely used in the corporate world to design training programs, it is still one of several available models of instructional designs. 

Another popular model is the SAM model, which is considered a popular alternative to ADDIE.

The abbreviation ‘SAM’ stands for Successive Approximation Model. Allen Interactions introduced it in 2012, and unlike the ADDIE model, SAM is a cyclical process that cuts short the time and cost of the program. 

It has several variations, but the most common feature of SAM is that it introduces iteration into the design process, including rapid prototyping and early addition of development and implementation in the process. 

Using SAM, you can eliminate one of the significant drawbacks faced while using the ADDIE model, i.e., the time-consuming nature of creating and implementing the program. The SAM allows you to reach the implementation phase faster. 

While the SAM does have appealing features like low cost and a faster process, early training programs based on the SAM often require several iterations and modifications to achieve the desired result. 

The ADDIE model and the SAM have unique features; hence, choosing between the two doesn’t need to be mutually exclusive. 

Selecting a model comes down to the organisation’s needs, the situation and the time available. The choice also depends on the team tasked with creating the training program. 

Pros vs. Cons of the ADDIE Model of Instructional Design

The advantages of ADDIE include being widely used and accepted as an effective model for human learning. It provides a solid foundation for other learning models, such as those based on cognitive psychology.

In courses designed using the ADDIE model, it is also easy to measure both time and cost.

On the other hand, the ADDIE model is also a rigidly linear process that requires following a specific sequence and does not have enough flexibility to adapt to unforeseen changes in the project. 

It is also time-consuming and costly and does not allow for iterative designs.

Frequently Asked Questions

What does the ADDIE acronym stand for?

The acronym ‘ADDIE’ stands for Analysis, Design, Development, Implementation, and Evaluation.

What is the importance of the ADDIE model?

The ADDIE instructional design model helps create a structured process to develop effective and efficient training programs. Every step in the ADDIE model helps ensure that the learning solutions effectively meet the learners’ needs. 

What’s the first step in implementing the ADDIE model?

The Analysis Phase is the first step in implementing the ADDIE model of instructional design. The analysis stage focuses on all the foundational factors involved in developing a course.

What is the final step in implementing the ADDIE model?

The final step in the ADDIE model of instructional design is the evaluation phase which involves measuring the efficiency and overall effectiveness of the course.

Where did the ADDIE model originate?

How is the ADDIE a helpful model?

The ADDIE model is outdated but still very useful. Although initially designed as a linear process, it is now being used iteratively. Once the five phases are completed, you can start over again. You can then use the analysis phase to improve your end product.

One Response

Great. Very perfectly articulated the ADDIE model. very informative with some latest research work on the topic. Keep on sharing such wonderful writeups.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

Subscribe our Newsletter

You might also like.

Top 9 Differences Between an Employee and Employer: A Comprehensive Guide

Enhancing Organizational Resilience through Effective Succession Planning

Unlocking the Power of Talent: A Comprehensive Guide to Talent Management in HRM

Become a certified human resource management professional, certifications.

• CHRMP Foundation
• HRBP Advanced
• HR Analytics
• CHRMP C&B Planning
• CHRMP Competency Mapping
• CHRMP Talent Acquisition
• CHRMP Talent Development
• CHRMP Tableau for HR
• HR Certification
• Competency Model
• For Fresh Graduates
• Career Change To HR
• Existing HR Professionals
• Employer/HR Manager
• Academy Login
• Academy Status Page
• Academy Roadmap
• Alumni Testimonials
• Know Your Facilitator
• International Affiliates
• Indian Affiliates
• Channel Partners
• CHRMP Perspectives
• Webinar Series
• Support Center
• Covid-19 HR Policy
• Coronavirus Resources
• CHRMP L&D GPT
• Cookies Policy
• Privacy Policy
• Terms of Service
• Refund & Cancellation Policy

Most Popular Location for Our Online HR Courses

Get Your Free HR Metrics Guide Book

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Front Public Health

Using Instructional Design, Analyze, Design, Develop, Implement, and Evaluate, to Develop e-Learning Modules to Disseminate Supported Employment for Community Behavioral Health Treatment Programs in New York State

Sapana r. patel.

1 The New York State Psychiatric Institute, Research Foundation for Mental Hygiene, New York, NY, United States

2 Department of Psychiatry, College of Physicians and Surgeons, Columbia University, New York, NY, United States

Paul J. Margolies

Nancy h. covell, cristine lipscomb.

3 Intrac Inc., Instructional Design and Learning Strategy, Reno, NV, United States

Lisa B. Dixon

Associated data.

Implementation science lacks a systematic approach to the development of learning strategies for online training in evidence-based practices (EBPs) that takes the context of real-world practice into account. The field of instructional design offers ecologically valid and systematic processes to develop learning strategies for workforce development and performance support.

This report describes the application of an instructional design framework—Analyze, Design, Develop, Implement, and Evaluate (ADDIE) model—in the development and evaluation of e-learning modules as one strategy among a multifaceted approach to the implementation of individual placement and support (IPS), a model of supported employment for community behavioral health treatment programs, in New York State.

We applied quantitative and qualitative methods to develop and evaluate three IPS e-learning modules. Throughout the ADDIE process, we conducted formative and summative evaluations and identified determinants of implementation using the Consolidated Framework for Implementation Research (CFIR). Formative evaluations consisted of qualitative feedback received from recipients and providers during early pilot work. The summative evaluation consisted of levels 1 and 2 (reaction to the training, self-reported knowledge, and practice change) quantitative and qualitative data and was guided by the Kirkpatrick model for training evaluation.

Formative evaluation with key stakeholders identified a range of learning needs that informed the development of a pilot training program in IPS. Feedback on this pilot training program informed the design document of three e-learning modules on IPS: Introduction to IPS, IPS Job development, and Using the IPS Employment Resource Book . Each module was developed iteratively and provided an assessment of learning needs that informed successive modules. All modules were disseminated and evaluated through a learning management system. Summative evaluation revealed that learners rated the modules positively, and self-report of knowledge acquisition was high (mean range: 4.4–4.6 out of 5). About half of learners indicated that they would change their practice after watching the modules (range: 48–51%). All learners who completed the level 1 evaluation demonstrated 80% or better mastery of knowledge on the level 2 evaluation embedded in each module. The CFIR was used to identify implementation barriers and facilitators among the evaluation data which facilitated planning for subsequent implementation support activities in the IPS initiative.

Instructional design approaches such as ADDIE may offer implementation scientists and practitioners a flexible and systematic approach for the development of e-learning modules as a single component or one strategy in a multifaceted approach for training in EBPs.

Background and Rationale for Educational Activity

A recent report by the Institute of Medicine Best Care at a Lower Cost: The Path to Continuously Learning Health Care in America ( 1 ), reported, “Achieving higher quality care at lower cost will require fundamental commitments to the incentives, culture, and leadership that foster continuous learning, as the lessons from research and each care experience are systematically captured, assessed, and translated into reliable care.” Central to the translation from research to practice and reliable care is training health-care providers in evidence-based practices (EBPs). In behavioral health care, training in EBPs often involves developing new clinical competencies. This training should take into account the context and needs of the practice community as well as strategies to facilitate adoption and implementation ( 2 ). Increasingly, training utilizes online modalities to expand its reach and efficiency, digital media to promote active engagement, shorter learning sessions to foster knowledge retention, and methods to demonstrate and practice skills that can be applied in the workplace ( 3 ).

Implementation science, a field dedicated to understanding targeted dissemination and implementation of EBPs and the use of strategies to improve adoption in community health-care settings, has guided the work of translating research to practice. Numerous frameworks in implementation science provide a menu of constructs that have been associated with effective implementation. Damschroder (Figure ​ (Figure1) 1 ) ( 4 ) combined 19 published implementation theories into the Consolidated Framework for Implementation Research (CFIR). The CFIR provides a menu of constructs that have been associated with effective implementation. The framework is organized into five domains: intervention characteristics, inner setting, outer setting, characteristics of individuals, and process . Under the inner setting domain, one key construct under the component readiness for implementation is access to knowledge and information . Access to knowledge and information is defined as the ease of access to digestible information and knowledge about the practice and how to incorporate it into work tasks. This is the function of training. It is purported that when timely on-the-job training is available, implementation is more likely to be successful ( 5 ).

Using an instructional design framework [Analyze, Design, Develop, Implement, and Evaluate (ADDIE)] as a systematic process to develop training in an implementation framework [Consolidated Framework for Implementation Research (CFIR)].

Although training is an important determinant of successful implementation ( 5 ), the field of implementation science lacks systematic approaches for the development of training that takes into account the learners’ needs, context, and optimal modalities for learning. Training in EBPs and their evaluation has been identified as a priority item on the National Institutes of Health research agenda (Program Announcement in Dissemination and Implementation Research in Health; https://grants.nih.gov/grants/guide/pa-files/PAR-16-238.html ) and a commonly used implementation strategy in implementation practice and research ( 6 ). Intervention or EBP developers are not likely to have expertise in instructional design and may miss the mark of engaging busy practitioners in training for several reasons. First, didactic approaches may not take into account the level of interest or needs of the practitioners. Second, traditional training approaches may not consider organizational factors (i.e., time available for provider training) also key to successful implementation ( 7 ). Third, how individuals learn and process information is evolving given our access to the Internet and technology. The field of implementation science may benefit from an ecologically valid approach to the development of learning experiences for training health-care practitioners.

Recent reports have pointed to the utility of instructional design in the dissemination and implementation of EBPs in behavioral health ( 8 , 9 ). The field of instructional design offers one model, Analyze, Design, Develop, Implement and Evaluate (ADDIE) ( 10 ), that takes into account learning theory, the learner’s needs and environment, and approaches to training practitioners in EBPs. The foundations of ADDIE are traced back to World War II when the U.S. military developed strategies for rapidly training people to perform complex technical tasks. The ADDIE model is used in creating a teaching curriculum or a training that is geared toward producing specific learning outcomes and behavioral changes. It provides a systematic approach to the analysis of learning needs, the design and development of a curriculum, and the implementation and initial evaluation of a training program ( 11 , 12 ). This model of developing training programs is particularly useful if the focus of the program is targeted toward changing participant behavior and improving performance. ADDIE is increasingly being adopted in industries such as health care ( 13 ). Recent studies have successfully adopted the ADDIE model to improve patient safety, procedural competency, and disaster simulation ( 14 , 15 ). It has also been effectively used in medical training and education to change practice behaviors in the management of various medical conditions ( 16 – 18 ).

The options for delivery and modalities used in training (e.g., mobile devices, webcasts, and podcasts) have expanded significantly in the last decade. With online learning technology, there is an opportunity to reach learners anytime and anywhere to provide performance support. One such example of an online learning technology is e-learning modules. e-Learning modules are self-paced lessons that enable the learner to read text, listen to narrated content, observe video scenarios, and respond to questions or prompts, in a multimedia format designed to maximize engagement and retention. Learning management systems (LMSs) host e-learning modules and capture learning metrics and performance. Learning analytics provided by LMSs enable the ability to track individual and group performance which may be used to provide feedback and support continuous learning in large systems of care.

In this report, we provide an example of the application of ADDIE in the development and evaluation of e-learning modules as one strategy among a multifaceted approach to the dissemination and implementation of the individual placement and support (IPS) model of supported employment in community treatment programs in New York State (NYS). Specifically, we (1) describe the application of an instructional design framework, ADDIE, in the iterative development of e-learning modules for IPS; (2) conduct a large-scale dissemination of the IPS e-learning modules throughout the state using an LMS; (3) evaluate learner reaction, self-reported knowledge, and practice change after IPS e-learning modules; (4) identify key barriers and facilitators to future IPS implementation using formative and summative ADDIE evaluation data and the CFIR.

Pedagogical Frameworks

We used three frameworks to guide the process of developing e-learning modules (ADDIE), identify determinants of future training and implementation (CFIR), and evaluate the IPS e-learning modules [Kirkpatrick model ( 19 )]. The ADDIE model consists of five phases, beginning with identifying key stakeholder needs, educational goals, and optimal methods of content delivery (analysis). This information was used to establish a design document for the training (design) that is vetted by key stakeholders prior to building the e-learning modules (development). After iterative refinement, e-learning modules were disseminated and evaluated using the Kirkpatrick model for training evaluation ( 19 ) (implementation/evaluation). Results from the formative and summative evaluations conducted during the ADDIE process, identified barriers/facilitators to implementation using CFIR domains (Figure ​ (Figure1). 1 ). Doing so allowed the IPS team to iteratively ensure sufficient attention to contextual variables, align with the larger conceptual and empirical implementation literature ( 9 , 20 ) as well as select strategies to build a multifaceted approach to IPS implementation.

Learning Environment

In November 2007, the NYS Office of Mental Health (OMH) and the Department of Psychiatry, Columbia University, established the Center for Practice Innovations (CPI) at Columbia Psychiatry and New York State Psychiatric Institute to promote the widespread use of EBPs throughout NYS. CPI uses innovative approaches to build stakeholder collaborations, develop and maintain providers’ expertise, and build agency infrastructures that support implementing and sustaining these EBPs. CPI works with OMH to identify and involve consumer, family, provider, and scientific-academic organizations as partners in supporting the goals of OMH and CPI. CPI’s initial charge was to provide training for the NYS behavioral health-care workforce. Given the size and geographical dispersion of this workforce, CPI turned to distance-learning technologies and e-learning modules ( 21 , 22 ). Distance technologies may offer cost-effective alternatives to typical training methods, and some evidence suggests that such technologies are at least as effective as a face-to-face training ( 21 ). CPI has collaborated with key stakeholders and content experts to create more than 100 e-learning modules to provide training for its initiatives. CPI’s online modules and resources require the use of an online learning platform, an LMS, that facilitates access to online training, event registration, and resource libraries for each initiative.

One of these initiatives, IPS, provides training and implementation support in an evidence-based approach to supported employment ( 23 ). Rates of competitive employment were low across NYS, with a competitive employment rate of 9.2% in 2011 prior to systematic IPS implementation (Patient Characteristics Survey data, 2011 obtained from https://www.omh.ny.gov/omhweb/statistics/ ). In response, OMH leadership identified supported employment as a key service in personalized recovery oriented services (PROS) programs, a comprehensive model that integrates rehabilitation, treatment, and support services for people with serious mental illness. The number of PROS programs in New York has increased significantly over the past decade: in 2017, 86 programs were serving 10,500 individuals. In order to reach these 86 programs statewide, the IPS initiative developed a series of three e-learning modules: Introduction to IPS, IPS Job Development , and Using the IPS Employment Resource Book . The module development team included an instructional designer, subject matter experts (SMEs), course developers, and a project manager.

Pedagogical Format: E-Learning Module Design Using Addie

Analysis: learning objectives.

In the analysis phase, the instructional problem was clarified, the instructional goals and objectives were established, and the learner’s environment, existing knowledge, and skills were identified. The module development team engaged in a discussion to identify the instructional problem and understand the expectations for performance after completing the modules. Because IPS had not been previously implemented in NYS, it was expected that learners’ existing knowledge and skills of IPS would be minimal. Formative evaluation via preliminary discussions with agency administrators, employment supervisors, and employment staff members in PROS programs included questions about learners’ experiences with and opinions about traditional vocational rehabilitation methods, attitudes about IPS principles (i.e., zero exclusion), awareness of or experiences with IPS, and expectations and attitudes about the likelihood of program recipients in their programs working competitively. These discussions revealed several needs: lack of understanding of the evidence for IPS ( CFIR: intervention ), discomfort with some IPS principles which are inconsistent with traditional approaches to vocational rehabilitation ( characteristics of individuals ), lack of knowledge about the specific skills and tasks involved in the model ( characteristics of individuals ), lack of familiarity with how to do job development and why it is important ( characteristics of individuals ), and the lack of tools that can be used in real-time meetings with potential employers ( implementation process ). These data informed the development of a curriculum for a pilot training program in IPS that consisted of in-person training, webinars, and on-site technical assistance. Through this pilot process, observations were made about learners’ strengths and additional training needs, and the PROS program environment. In addition, program recipients’ (adults diagnosed with serious mental illness, living in the community, many with histories of hospitalizations and treatment) employment needs (e.g., consistent with individuals’ personal strengths and interests), part-time for many, easily accessible with public transportation ( outer setting ) supported another cycle of modifications to the IPS curriculum and informed decisions about pedagogical format. As the initiative required scalability across the state of New York, it was determined that e-learning modules would be an important resource-efficient implementation strategy.

The design phase established learning objectives, exercises, content, lesson planning, and media selection via a design document, which served as the blueprint for building the training program. The instructional designer gathered feedback from the analysis phase and resources on the topic provided by SMEs (e.g., books, research publications, information available online) and identified content to support the learning objectives (Table ​ (Table1) 1 ) for all three IPS e-learning modules. The module development team designed a 10-item knowledge quiz and a 10-item level 1 reaction survey consisting of both closed- and open-ended questions. Iteratively, the instructional designer presented design documents for review and feedback from the module development team. An example design document for the IPS Job Development module is provided in the Figure S1 in Supplementary Material.

Learning objectives for individual placement and support (IPS) modules.

Development

During the development phase, the course developer received the reviewed design document and used an authoring tool software to create multimedia e-learning modules according to the design document. During this phase, the IPS modules were animated using video, graphics with narration, knowledge checks, and photographs. Formative evaluation from the analysis phase led to the development of a tool, the Employment Resource Book ( 24 ), that could be utilized by key stakeholders (providers, supervisors, and recipients) during any phase of employment (e.g., considering work, actively seeking employment, maintaining employment), and one module was developed to provide guidance about using this resource. The IPS training was built into three short e-learning modules to reflect learners’ time availability and attention span during the workday, then tested in prototype with the module development team and revised.

Implementation

During the implementation phase, e-learning modules were uploaded to the CPI LMS for usability testing. During usability testing, the module’s functionality is evaluated prior to training implementation. For example, the module development team tested whether videos play and navigation works (e.g., next buttons and links to additional resources) on a variety of web browsers and devices. Feedback from the usability testing phase is used to fix errors in navigation and improve user experience ( 25 ). After usability testing issues were addressed, the modules were ready for implementation.

When the IPS initiative began, the NYS-OMH Rehabilitation Services Unit sent an official email communication strongly encouraging PROS program providers and supervisors to participate in the training offered by the CPI IPS initiative. Further, each PROS program supervisor received an email, alerting them that the new IPS e-learning module was available in CPI’s LMS. Through the LMS, PROS program participation in the modules was tracked, and completion could be monitored by PROS programs and NYS-OMH.

We applied quantitative and qualitative methods as part of formative and summative evaluation in the ADDIE process. Formative evaluations consisted of qualitative feedback received from recipients and providers during early pilot work, which identified training needs. The summative evaluation consisted of quantitative and qualitative data and was guided by the Kirkpatrick model for training evaluation ( 19 ). The four levels of evaluation are (1) the reaction of the learner about the training experience, (2) the learner’s resulting learning and increase in knowledge from the training experience, (3) the learner’s behavioral change and improvement after applying the skills on the job, and (4) the results or effects that the learner’s performance has on care provided. For this report, we focus on the first two levels, specifically, the level 1—reaction of the learner including training experience, self-reported knowledge acquisition, and self-reported practice change through a survey and level 2—resulting knowledge through post-module quizzes.

To keep the learner experience seamless, a decision was made to embed the knowledge quiz, assessing knowledge of IPS model-related concepts, skills, and tools, within each module. In order for the module to be marked as completed, learners are required to answer at least 80% of the knowledge items correctly, which satisfies continuing education accreditation requirements. Learners are able to retake the quiz as many times as needed to meet this criterion score. Once the module is completed, the learner is prompted to complete the level 1 survey. The level 1 reaction survey was based on learning objectives set forth in each e-learning module, accreditation requirements, and example questions from Kirkpatrick level 1 ( 19 ). Questions included rating the module overall, if it met stated learning objectives, if the information presented was new to the learner, and questions about module-specific self-reported knowledge and practice change. In addition, three open-ended questions were included: What could we improve? What do you like the most about this module? and Where do you think you might use what you learned in this module?

The NYS Psychiatric Institute Institutional Review Board determined that this evaluation did not meet the definition of human subject research.

Using IBM© SPSS© Statistics Version 24, we applied descriptive statistics to quantitative level 1 summative evaluation data. For the qualitative formative and summative evaluation data, we employed a thematic analysis to identify themes within the open-ended question data ( 26 ). Two coders reviewed the open-ended question data independently to identify codes and develop an initial code list. The coders combined codes into overarching themes and met to review and label them. Coders met twice to discuss discrepancies and achieve consensus on key barriers and facilitators within the CFIR framework. We report on those themes that were raised by at least 10% of the sample.

We describe the inputs and outputs during each phase of IPS module development using ADDIE in Table ​ Table2. 2 . Formative evaluation during each stage of ADDIE allowed for the iterative revision of the content for each module and the identification of needs for subsequent modules. Feedback received from the evaluation of the first module led to the development of the second module (i.e., desire to learn more about job development) and to the development of the Employment Resource Book including the associated third module (i.e., desire to be better equipped to deal with common challenges).

Using Analyze, Design, Develop, Implement, and Evaluate (ADDIE) to develop individual placement and support modules.

Summative evaluation examined the impact of the IPS training modules and assisted in the identification of barriers and facilitators for IPS implementation in the future. Table ​ Table3 3 summarizes level 1 evaluation data for all three IPS modules. Learners’ background and experience varied considerably across programs. Many were rehabilitation counselors, social workers, and some had non-behavioral health backgrounds. Learners rated all three modules highly (mean range: 4.4–4.5 out of 5). Learners also indicated that the modules presented new information and met their stated learning objectives (mean range: 4.3–4.4 out of 5). Similarly, learners’ self-report of knowledge acquisition was high (mean range: 4.4–4.6 out of 5). About half of learners indicated that they would change their practice after watching the modules (range: 48–51%). All learners who completed the level 1 evaluation demonstrated 80% or better mastery of knowledge on the level 2 evaluation embedded in each module.

Level 1 data from all three individual placement and support (IPS) modules.

a N = 523 .

b N = 312 .

c N = 127 .

d Likert scale: 1-strongly disagree to 5-strongly agree .

Open-ended question themes and related CFIR domains from these e-learning modules helped identify additional implementation support needs to be addressed by the multifaceted approach to implementing IPS (i.e., statewide webinars, regional online meetings focusing on special topics such as IPS fidelity and supervision, an IPS library with tools to help IPS implementation, and individualized program consultations that focus on addressing implementation challenges and enhancing provider competence). Themes from the open-ended questions for all three IPS modules are described using the CFIR in Table ​ Table4. 4 . These themes related to three CFIR domains: outer setting, inner setting, and implementation process. They provided information on how the modules were acceptable, what the future learning needs are, and how the information learned will be used in everyday practice.

Themes and CFIR domains from level 1 survey open-ended questions for individual placement and support (IPS) modules.

This report provides one example of how an instructional design approach may be applied to the development of e-learning modules as one strategy in a multifaceted approach to the implementation of IPS supported employment for community program providers in a large state public behavioral health system. Through iterative development, we applied the ADDIE model to develop a series of e-learning modules for IPS. Using an LMS, these modules were disseminated and evaluated by PROS program providers throughout NY state. Results from both level 1 and level 2 evaluations indicate that the ADDIE model was successful in improving practitioner knowledge. In addition, learners received the e-learning modules favorably, rating them highly overall and noting that they met stated learning objectives and presented new information. Throughout the development process, data from the e-learning modules were described using the CFIR to identify needs that led to additional e-learning modules as well as strategies for subsequent implementation supports through a learning collaborative statewide ( 27 ).

The ADDIE model and CFIR were used as complementary approaches in the development of e-learning resources for training providers in an EBP. Our experience in this process produced several lessons learned and recommendations for implementation researchers and practitioners. The analysis phase of the ADDIE model required assessment of multistakeholder needs and context early on in the process of developing training. We recommend taking the time to assess and include end users and recipients to shape and increase the ecological validity of the training. In addition, the use of the CFIR domains allowed us to anticipate barriers and map future implementation strategies. During the design process, the establishment of clear and measureable learning objectives was important and facilitated focus and evaluation of knowledge and skill acquisition. We recommend the a priori assembly of e-learning module development teams to work with the instructional designer and establish an efficient process for the review of training content and format through weekly iterative review meetings during the design and development stages. Although the ADDIE process points to the introduction of the learning platform (e.g., website, LMS) at the implementation stage, we would recommend that the team with technical expertise (i.e., in our case, the courseware developers and LMS administrators) be introduced earlier in the process during the development stage. This is crucial to the feasibility and usability of the end product. Once implemented, we recommend a scheduled monthly review of the evaluation data that is being collected as learners participate in the e-leaning modules. This information will identify any needed revisions to the training content, the need for future content development, and barriers and facilitators for future implementation.

This article reports on the development of e-learning modules that were one part of a larger implementation effort in a state system. This implementation was not a part of a rigorous research evaluation. Limitations of this report include inability to formally assess pre–post knowledge, practice and readiness for IPS implementation using validated scales based on accepted standard in the literature, variation in sample sizes for the e-learning modules precluding examination of a stable cohort of learners over time, and the inability to directly assess the specific impact of these e-learning modules on employment outcomes apart from other elements of the entire initiative. Notably, only half of the providers who completed the evaluation noted an intention to change their practice, and we did not have the capacity to assess practice change at the individual provider level at this stage of IPS implementation (level 3). However, in our subsequent work ( 27 ), program fidelity assessments using established measures demonstrated improvement over time, suggesting that level 3 provider practice change and fidelity self-assessed by program sites are shown to be associated with higher employment rates (level 4), which are sustained over time ( 28 ). Future research may focus on more rigorous evaluation of knowledge, practice change, mixed-method assessment of how the content from e-learning modules influences practice, and the essential role of care recipients in helping to design training within implementation efforts.

From adoption to sustainability, implementation science focuses on strategies to promote the systematic uptake of research findings into routine practice. Successful implementation relies on iterative, interacting activities that follow a systematic process for strategy development. In the case of training as an implementation strategy, instructional design offers a systematic and iterative process. First, it applies instructional theory to the development of training regardless of subject matter. Second, it identifies fundamental elements of the learners’ needs and real-world setting factors in addition to the EBP being implemented. Third, it creates accountability to align training content with measurable learning objectives and assesses learner knowledge and skill acquisition based on content. Lastly, it engages multimedia novel approaches in the development of educational and training resources.

Compared to more intensive approaches to training and workforce development, the development of e-learning modules informed by an instructional design approach provides implementation science the opportunity to scale and support training at the level of knowledge and skill acquisition for a range of EBPs. These modules can be used either as stand-alone or as part of blended learning activities and implementation supports as in the IPS initiative. Another example, in the case of complex, multi-component intervention or model of care, is CPI’s work with Assertive Community Treatment, where instructional design is used to develop e-learning modules as a first step in a blended learning training curriculum for practitioners in a state system ( 29 ). As such, there is increasing interest in examining the effect of an instructional design approach to training in behavioral health, especially for large systems of care.

Instructional design approaches such as ADDIE may offer implementation scientists and practitioners a flexible and systematic guideline for the development of e-learning modules as a single component or one strategy in a multifaceted approach for training practitioners in EBPs. In this way, this approach facilitates the translation between science to practice that takes into account the context of the learner and leverages technology for expanded reach, both promising approaches for workforce development and a learning health-care system ( 1 , 30 ).

Author Contributions

SP, CL, and LD conceived the study and conceptual framework. SP and NC managed data and analyses. SP, PM, NC, and CL contributed to writing the manuscript with feedback and supervision from LD.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. The reviewer AR and the handling Editor declared their shared affiliation.

Acknowledgments

SP is a fellow of the Implementation Research Institute (IRI), at the George Warren Brown School of Social Work, Washington University in St. Louis, through an award from the National Institute of Mental Health (R25 MH080916) and the Department of Veterans Affairs, Health Services Research & Development Service, Quality Enhancement Research Initiative (QUERI).

Supplementary Material

The Supplementary Material for this article can be found online at https://www.frontiersin.org/articles/10.3389/fpubh.2018.00113/full#supplementary-material .

IPS job development design document.

An Introduction to the ADDIE Model for Instructional Designers

If you’ve been around e-learning a little while, you’ve probably heard of ADDIE—the most commonly used instructional design model training designers use when crafting learning experiences. The acronym stands for: Analysis, Design, Development, Implementation, and Evaluation.

The five phases of the ADDIE model are designed to help guide you (and your team) through the course design process. Essentially, it’s a roadmap for building training that ensures learners meet specific objectives. New designers have a tendency to want to jump directly into development (I totally get it; that’s the fun part!), but it’s important to understand the bigger picture before making that leap. Let’s take a closer look at what each phase of ADDIE entails.

According to the ADDIE model, the first thing you should do when you’re handed a new training project is a detailed analysis. What do you need to analyze? Three of the most critical analyses carried out by instructional designers are:

• Training Needs Analysis: This should be the first type of analysis you complete because it identifies whether the training is needed at all. This analysis identifies what performance improvements are expected and how they’ll be measured, which is critical in identifying whether training has been successful. Read more: How to Do an E-Learning Needs Analysis .
• Audience Analysis: Once you confirm that training is needed, it’s time to analyze your learners. Knowing key demographics and background information about your learners will help you identify the information they need to know and the best way to present it to them. Read more: How to Do an E-Learning Audience Analysis .
• Task Analysis: Now that you know what your course is about and who will be taking it, it’s time to take a close look at the specific processes and tasks you’ll be training learners on by breaking them down into step-by-step chunks. Read more: How to Do a Task Analysis Like a Pro .

Once you’ve completed these analyses, you will have a much better idea of the who, what, where, and why of your e-learning. As a next step, it’s a good idea to compile these findings into your first deliverable: a project plan. Read more: How to Plan E-Learning Courses Like a Pro .

With the analyses out of the way, you’re ready to start development, right? Wrong! Don’t skip the first “D”! Starting to develop your course without completing the design phase is like building a house without a blueprint. It makes much more sense to start with a clear plan of how everything in the course will be laid out and how the text, multimedia, and navigation will fit together. The deliverable you create depends on your time, budget, resources, and what you’ve outlined in your project plan. Typically, one of two deliverables comes out of the design phase:

• Storyboard: This document lays out the elements of the course that will appear on each slide. This may include text, imagery, and narration script. Deciding what to include in a storyboard depends on the project. For example, if your project includes audio narration, you’ll need to include a script with your storyboard. If you’re building a storyboard that you’ll hand off to a developer, you’ll need to add detailed notes for them. Read more: Storyboards for E-Learning: What to Include?
• Prototype: This typically includes sample slides to test and identify whether specific features or concepts work. The prototype lets a stakeholder get a feel for how the course will look and function before developing the entire course and all of its features. Read more: E-Learning: Storyboard vs. Prototype .

Once you’ve got your blueprint for your course, you’re ready to jump into the fun part: development!

This is the phase where you (finally!) get to build out your e-learning content in an authoring app. The development part of the ADDIE process typically contains two sub-tasks:

• Content Creation: In this phase, you choose and add the final graphics, multimedia, colors, and fonts to make your course look polished and professional. You’ll also use your authoring app to build out activities, quizzes, interactions, and functional navigation to create an engaging course. Read more: The Basics of E-Learning Course Creation Apps .
• Testing: Once you’ve created your content, you need to test it. Things that need to be tested and reviewed include spelling, grammar, learning objectives, navigation, and flow. Testing is typically done during the development process instead of after, so the developer can make changes as testers identify problem areas. Read more: Top 4 Tips for E-Learning Quality Assurance (QA) Testing .

Once your course is fully developed and thoroughly tested, you’re ready to share it with your learners. Not sure how to do that? Check out this article to find out more about your options: How to Share E-Learning Courses with Learners .

When you progress to the evaluation phase, you need to go back to the very first phase of the ADDIE process, in which you (hopefully!) completed a training needs analysis. During that phase, you identified specific performance improvements that your training would address, as well as how to measure those improvements.

The evaluation phase is where the rubber meets the road: Did your training result in the real, measurable performance improvements you identified in your needs analysis? While the learners’ opinions and feedback about the e-learning matter, it’s critical to ensure your training achieved the goals you set at the start. Want to learn more? Check out these articles:

• Post-Course Evaluations: What E-Learning Designers Need to Know
• How to Measure the Satisfaction of Learners Taking Your Online Courses

And there you have it! Those are the five phases of the ADDIE model. Having a solid foundation of each phase will ensure you end up with a high-quality course that meets the needs of your learners.

Interested in learning about other instructional design models? Here are a few related articles:

An Introduction to SAM for Instructional Designers

• An Introduction to Bloom’s Taxonomy
• Measure the Effectiveness of Your E-Learning Course with Kirkpatrick’s 4 Levels of Evaluation
• The Presentation/Application/Feedback (PAF) Model

An Introduction to Instructional Design

Follow us on Twitter and come back to E-Learning Heroes regularly for more helpful advice on everything related to e-learning. If you have any questions, please share them in the comments.

Related Content

17 Comments

• Melinda Abrahamson

• Nicole Legault

• Alexander Salas

I think that SAM is a very direct way to get the Stakeholders, SME's and developers on the same page. Everyone is basically notified that while the project is in development, their timely feedback is important to creating the best module, with the least amount of setbacks, almost like creating constantly evolving design period for the entire project. Working on the entire project (SAM) in a program like Articulate, instead of a module at a time (ADDIE), is way more efficient, because changes can be made almost instantaneously. It seems to give the Stakeholders and SMEs more of a sense of ownership in the project, too, urging them to be more responsive and vocal on the things that matter without having the financial burden of the setback. (By the way, I agree about the bad wrap Pow... Expand

• Angie Carter

• Mike Hunsberger

• Section 2: Home
• Developing the Quantitative Research Design
• Qualitative Descriptive Design
• Design and Development Research (DDR) For Instructional Design

Overview of Design and Development Research (DDR) For Applied Doctorate Students in the Instructional Design Program

Types of design and development research, 3 stages in design and development research, data collection methods and sources of data in ddr.

• Qualitative Narrative Inquiry Research
• Action Research Resource
• Case Study Design in an Applied Doctorate
• SAGE Research Methods
• Research Examples (SAGE) This link opens in a new window
• Dataset Examples (SAGE) This link opens in a new window
• IRB Resource Center This link opens in a new window

The purpose of this quick guide is to assist Applied Doctorate students in the Instructional Design Program in determining the best methodology and design for their Applied Doctorate Experience (ADE) dissertation. The guide covers intended target audience, an overview of Design and Development Research (DDR), types of DDR research including product, program, tool research, and model research, 3 stages providing alignment of DDR with NUs Applied Doctoral Record (DDR) deliverables, examples of problem, purpose, and research questions for DDR research, and suggested references. 

Target Audience: Doctoral Students in Instructional Design in the ADE program

This quick reference guide will aid doctoral students in instructional design challenged with deciding on what type of applied research study they want to do for their dissertation.

Overview of Design and Development Research (DDR) Methods

At the core of the instructional design and instructional technology and media field, is the design, development, implementation and evaluation of instructional products, tools, programs, models, and frameworks.  In many ways DDR is like Action Research (Goldkuhl, 2012), however, there are many differences. DDR research allows instructional designers a pathway to test theory, models, and frameworks and to authenticate practice. The focus of DDR is on the use, design, development, implementation, and evaluation of products, tools, programs, and models using instructional design models and frameworks. Richey and Klein (2007) defined DDR as “the systematic study of design, development, and evaluation processes with the aim of establishing an empirical basis for the creation of instructional and non-instructional products and tools and new or enhanced models that govern their development” (p. xv). Often the models and frameworks are validated and/or further developed and enhanced through the DDR. DDR is applied research. An area of DDR research that is particularly applicable to ADE students is the creation, implementation, and evaluation of one or more artifacts, such as products, tools, models, new technologies, and learning objects that will aid in solving a complex problem in practice that can be addressed through human imagination, creativity, engagement, and interaction (Ellis & Levy, 2010). These types of problems are found in K-12 education, higher education, corporations, not-for-profits, healthcare, and the military. 

• Example Design and Development Research

The field of DDR is constantly evolving and expanding as technology and media are changing at exponential rates.  Richey and Klein (2007) in their seminal work divided DDR into two major categories:

• Product and Tool Research and
• Model Research.

Table 1 provides a summary of common designs used in DDR. Most DDR work falls under the qualitative research category of qualitative case study, however, methodologies such as quantitative and mixed method have been used as well as other qualitative designs, including Delphi.

Product, Program, and Tool Research

Ellis and Levy (2010) asserted that DDR must go beyond commercial product development by determining a research problem, based on existing research literature and gaps in the literature that researchers assert must be studied to add to the instructional design knowledgebase.

Product and Tool Research can be further divided into:

• Comprehensive Design and Development Projects covering all phases of the instructional design process,
• Specific Project Phases (such as those in the ADDIE model: Analysis, Design, Development, Implementation, and Evaluation), and
• Design, Development, and Use of tools (Richey & Klein, 2007).

Model Research

Instructional designers and instructional technologists have focused on model research since the emergence of the field.

Model research can be broken into three types:

• Model Development,
• Model Validation and

Model development can focus on a comprehensive model design or on part of a process. Model validation research uses empirical processes to prove the effectiveness of a model in practice. Finally, model use research addresses usability typically from the perspective of instructional designers and stakeholder experts.

3 Stages in Design and Development Research for the ADE Doctoral Student’s Dissertation

NU doctoral students in the Instructional Design Program can use one of the various types of DDR research to complete their doctoral dissertation using the NU ADE template. There will be three stages in this process and in each stage the student will have one or more deliverables using the NU template and posting in the ADR on NU One.

Stage 1: Design and Development Research aligned with the NU ADE Template Process

• Identify a research worthy problem which is expressed by researchers in peer reviewed research literature. Ask yourself, what is going wrong? What do researchers say is known about the problem? And what is needed to be known to address the problem?
• Describe the purpose of your research ensuring that it aligns with your problem statement. In the description state your methodology and design and which DDR type of research you will do. Be sure to include a description of your target population (audience), the size of your sample and the sampling strategy you will use to access your sample. What permissions do you need? Site permission? Other IRB permission?
• Write your research questions to align with your problem and purpose statements.
• Complete Section 1 of your Applied Doctoral Experience (ADE) template securing all necessary approvals in the Applied Doctoral Record (ADR).
• Needs Assessment
• Measurable Goals and Objectives
• Sample size and Access to the sample
• Sampling strategy
• Content analysis (course, program, product, or tool descriptions)
• Technology and media analysis/selection
• Learning management system(s)
• Asynchronous
• Synchronous
• Evaluation Plan
• Complete Section 2, Proposal Draft, Proposal for AR, and Final Proposal of the ADE securing all necessary approvals in the ADR.
• Submit Proposal and IRB Application to secure IRB approval.

Stage 2: Design and Development Research aligned with the NU ADE Template Process

After receiving IRB approval of your ADE Proposal, it is time to design, develop, test, validate, and/or evaluate your artifacts. Below are example steps:

• Review and Finalize Design Document
• Recruit Expert Participants, if required
• Recruit Artifact User/Participants, if required
• Lesson plan or syllabus
• Instructional strategies and activities
• Participant materials
• Trainer materials
• Storyboards and scripts
• Other media
• Create model, tool, product, or program.
• Validate model, if required
• Evaluation plan (Kirkpatrick Levels 1, 2, 3, 4)
• Alpha test, Beta test, Pilots.
• Rapid Prototyping
• Participant reaction
• Trainer/facilitator reaction
• Were Design Goals met?
• Were Design Objectives met?
• Revise artifact(s), Retest, if necessary.

Stage 3: Design and Development Research aligned the NU ADE Template Process

Complete Section 3 of the ADE template presenting the study findings, conclusions, and implications. Next pull all three sections into a dissertation manuscript for approval in the ADR.

While DDR covers a wide variety of approaches, most doctoral students in the ADE program will find case study to be the preferred design. To strengthen trustworthiness of the data, multiple sources of data will typically be used.  Using multiple sources of data is called triangulation in research. Figure 1 shows examples of sources of data for DDR.

The goal is to create, use, and/or validate New Artifacts by collecting and analyzing various sources of data including:

• Existing artifacts,
• Expert individual and focus group interviews,
• Participant/user individual interviews, talk aloud-think aloud interviews, focus group interviews,
• Research observation and participant observation,
• Evaluation, Kirkpatrick Levels 1-4, and
• Needs assessment and design documents.

The new artifacts may be lesson plans, student guides, facilitator/teacher guides, learning objects, tools, models, programs, and/or products.

Figure:  Sources of Data in DDR

Ellis, T.J. & Levy, Y. (2010). A guide for novice researchers: Design and development research methods. Proceedings of Informing Science & IT Education Conference (InSITE) 2010, pp. 108-118. http://proceedings.informingscience.org/InSITE2010/InSITE10p107-118Ellis725.pdf

Goldkuhl, G. (2012). From Action Research to Practice Research. Australasian Journal of Information Systems, 17 (2). https://doi.org/10.3127/ajis.v17i2.688

Richey, R. C. & Klein, J. D. (2007). Design and Development Research. Routledge

• << Previous: Qualitative Descriptive Design
• Next: Qualitative Narrative Inquiry Research >>
• Last Updated: Jul 28, 2023 8:05 AM
• URL: https://resources.nu.edu/c.php?g=1013605

© Copyright 2024 National University. All Rights Reserved.

Privacy Policy | Consumer Information

Accessibility Links

• Skip to content
• Skip to search IOPscience
• Skip to Journals list
• Accessibility help
• Accessibility Help

Click here to close this panel.

Purpose-led Publishing is a coalition of three not-for-profit publishers in the field of physical sciences: AIP Publishing, the American Physical Society and IOP Publishing.

Together, as publishers that will always put purpose above profit, we have defined a set of industry standards that underpin high-quality, ethical scholarly communications.

We are proudly declaring that science is our only shareholder.

Using the ADDIE model to develop learning material for actuarial mathematics

E Widyastuti 1 and Susiana 1

Published under licence by IOP Publishing Ltd Journal of Physics: Conference Series , Volume 1188 , The Sixth Seminar Nasional Pendidikan Matematika Universitas Ahmad Dahlan 2018 3 November 2018, Yogyakarta, Indonesia Citation E Widyastuti and Susiana 2019 J. Phys.: Conf. Ser. 1188 012052 DOI 10.1088/1742-6596/1188/1/012052

Article metrics

11261 Total downloads

Share this article

Author e-mails.

[email protected]

Author affiliations

1 Universitas Negeri Medan, Jl. Williem Iskandar Pasar V Medan Estate, Indonesia

Buy this article in print

The research is aimed to describe; (1) the development procedure of Actuarial Mathematics learning material with ADDIE model; (2) Validation of learning material using the ADDIE model for Actuarial Mathematics. The research method used is research and development using ADDIE Models. The instrument were used observation and quesionare. The data were analyzed by descriptive qualitative and descriptive quantitative. The results showed that (1) the process of designing and development of the material teachings has followed the five steps in ADDIE model such as analyze, design, development, implementation, and evaluation. (2) The result of the content expert's validation was falling into agreement category, that of the instructional design expert's validation was agreement, that of the instructional media expert was agreement. There were some comment that given by expert about module. The average of student's quesionare were falling into good category.

Export citation and abstract BibTeX RIS

Content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence . Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.

The ADDIE Model: The perfect formula for creating e-learning content

• | August 6, 2024
• | Elizabeth Aguiar Chacón
• E-learning content creation

Table of contents

There can be little doubt that technology has revolutionized the way we access knowledge and pursue professional development over recent years. Online learning is perhaps one of the best examples of this. The expansion of the internet, as well as the acceleration of mobile technology has led the way to a much more flexible, dynamic and agile type of training, where students have the chance to learn at their own pace and from any location.

However, this transformation is also bringing considerable challenges to the forefront, and designing effective learning experiences in virtual environments is one of them. Educators and instructional designers face the problem of not only having to generate attractive learning materials, but also ensure that they are effective enough to meet the proposed educational objectives. Fortunately, there are several different approaches and frameworks available to help facilitate this process. In this article, we’re going to talk about one of the best known – the ADDIE model.

What is the ADDIE model and what is it used for?

The ADDIE model is a creative framework commonly used in instructional design . It is a process consisting of five different phases: analysis; design; development; implementation; and evaluation . Together, they form a tightly structured approach to creating, implementing and evaluating educational resources.

The range of training materials that can be created with this instructional design model is extensive, and includes e-learning courses, interactive training modules, manuals and study guides, educational games, assessments and questionnaires, videos and tutorials, and visual materials. Through this simple but effective process, it becomes possible to easily identify the learning needs of students, as well as determine the main objectives of their training, which makes it easy to design activities that aim towards these goals.

History and origin

Although the ADDIE model was developed in the 1970s , it has steadfastly remained one of the most important references in instructional design since that time. It was initially developed within the State University of Florida, and its chief purpose was to provide assistance to the American army.

The model formed part of a broader effort to standardize and systematize the process of creating training materials for the military. Afterwards, the ADDIE model continued to evolve and became increasingly popular in academia thanks to such influential texts as “The systematic design of instruction” by William W. Lee and Robert E. Gagne (1988).

Key features of the ADDIE model

Although the ADDIE model we use today is markedly different from the one used in previous decades, it remains a highly popular choice in the design of training experiences. This is mostly due to its organized structure, flexibility, focus, versatility, and various other qualities. Let’s look at some of the most important of these:

A systematic and modular structure

ADDIE provides a very clear structure that breaks down the instructional design process into the 5 phases outlined above . The value of this segmentation is that it allows designers to follow a systematic and organized approach throughout.

Focuses on needs

The model focuses closely on the needs and objectives of both students and organizations. This means that any training program designed with this methodology is sure to be as relevant as it is effective.

Goal-oriented

Another important feature of the ADDIE model is its focus on achieving goals . In fact, the metric that determines the success of any training program developed with this methodology is whether or not it meets its proposed objectives.

An iterative approach

Although it divides itself into sequential phases, the ADDIE model is iterative. This means that it can be consistently improved by observations and feedback , allowing for revisions and adjustments at each stage based on continuous evaluation.

Pros and cons of the ADDIE model

Although the ADDIE model is widely used in both formal and informal education, it has pros and cons just like any other methodology. Let’s take a look at some of them:

Advantages of the ADDIE model

• Flexibility: Although it follows a set order of phases, the ADDIE model does allow for adaptations and revisions to be made to the process based on both the needs of the project and the results obtained in previous phases.
• Adaptability: As an instructional design model, it can be used across multiple different learning environments, disciplines, and industries . It is also useful for both large-scale projects, as well as for individual or group learning.
• A focus on continuous improvement: The model is built on the concept of continuous improvement through feedback, which allows the course to be improved and optimized over time.
• Ease of collaboration: The model brings together and involves several different parties , including teachers, designers, subject matter experts, and even students.
• A strong evaluative component: It’s important to remember that the ADDIE model has a significant evaluation component, effectively allowing organizations to measure the impact of their training. This is useful for identifying gaps in training and suggesting improvements.

Disadvantages of the ADDIE model

• The sequential approach can be time-consuming , especially for larger or more complex projects.
• It demands a certain quantity of resources , such as trained staff members and specific technological tools.
• It focuses mostly on the design and development of content , as opposed to interaction between students and instructors.

Phases of the ADDIE model

As we’ve already mentioned, the ADDIE model consists of five key phases: analysis; design; development; implementation; and evaluation. Each stage has a crucial part to play in the wider process and, due to their sequential nature, it is always necessary to have completed one stage before moving on to the next. Let’s take a closer look at the five stages:

The initial phase of the process hinges on collecting information . In order to do this, you first need to identify the needs or problem that you wish to resolve. For example, you may be looking to develop the skills that would allow a team to become completely digital. In such a case, having first identified the problem, you would then need to decide on the most effective training approach to solve it.

At this stage, answering a few simple questions may help you towards an answer:

• What is the purpose of the training?
• Why does it need to be carried out?
• What are the desired outcomes?

You will also need to identify the target audience of your training at this point, taking into account their needs, expectations, and learning styles . From there, you can move on to considering topics and content, as well as the tools you intend to use. This is also the right time, if necessary, to think about the learning modality (in-person, hybrid, or online) you wish to implement . In short, you need to map out all of the variables involved before actually starting to design the training materials.

As its name suggests, this is the stage where all the information collected in during analysis is converted into a learning design . Here, designers must set about creating a plan for the program, which will serve as a guide throughout the entire development process. This plan must include the key learning objectives, as well as cover the strategies to be used during training, any relevant assessment methods, and finally the specific way in which the content will be delivered. These should all have been settled upon in the previous stage.

Goal-setting techniques such as SMART can be helpful at this point. The important thing here is to design strategies that help students to achieve these goals through activities, evaluations, exercises, discussions and more.

Development

In the third phase, the plans, diagrams or storyboards that have been settled upon in the design phase – along with any relevant learning objectives and strategies – will be used to actually create courses. During the development phase, the training program becomes a tangible reality, whether produced internally or through an external provider.

The main part of this phase is the creation of resources in line with decisions made during the analysis and design stages. This is where the instructional designer comes into play, seeking to ensure that the learning materials they produce fit the design specifications as well as meet the needs identified during analysis. Once the material has been produced, common errors (such as spelling and coherence) can be checked for within any text, and throughout the entire navigation experience more generally.

Implementation

Implementation covers the whole process of actually delivering and managing training. This phase includes communications, logistics, data collection and, of course, the training itself.

During the implementation stage, the previously developed materials are put into practice, and delivered to students. This process may include uploading content to the e-learning platform, preparing the necessary infrastructure, and finally training the instructors and any other facilitators. It’s also crucial to ensure that learners have access to the resources they need, and that the operation of the course can be properly monitored to identify and settle any technical or logistical issues. In all, an effective implementation should ensure that every learner can properly interact with the training materials and actively participate in the learning process.

The final phase is evaluation. This is a key phase, making it possible to measure the effectiveness of the course and collect data on learner performance and progress. The stage can be divided into both formative and summative evaluation. Formative evaluation takes place throughout the development and implementation of the course, to provide live, continuous feedback and allow for real-time adjustments. Summative evaluation is carried out at the end of the course, and aims to measure the achievement of learning objectives and levels of learner satisfaction. The results can then be used to improve and optimize the design and delivery of the whole course, ensuring that the quality of training is continuously improved.

How to implement the ADDIE model in corporate training

The ADDIE framework offers the perfect structure for developing effective, efficient training programs. Here’s how you can use it to design your own corporate training programs:

• The analysis phase lets you identify desired skills as well as any knowledge gaps, so you can design programs that address specific learner needs.
• During the design phase, you’ll establish clear learning objectives and set pedagogical strategies in line with corporate goals.
• The development phase will allow you to create customized training materials – such as e-learning modules, simulations and others – that make learning possible.
• During the implementation phase, you’ll deliver the programs you’ve produced to employees, integrating them into your corporate learning platforms and providing easy access to all the necessary resources.
• Finally, in the evaluation phase, you’ll have the chance to measure the impact of training through formative and summative evaluations. The data collected here will help you to continuously improve the process, and demonstrate return on investment in training.

Alternatives to the ADDIE Model

Although the ADDIE model is widely used in the world of instructional design, there are several alternatives that can be just as effective, depending on the context and specific needs of the project:

1. The SAM Model (Successive Approximation Model)

The SAM model , developed by Michael Allen, is an agile, customer-focused methodology that relies on rapid prototyping and continuous improvement. Unlike the sequential approach favoured by ADDIE, SAM emphasizes collaboration and quick, constant adaptation through short, repeated cycles of design, development, and review. This allows designers to respond quickly, making continual changes to improve the final product based on the feedback they receive.

2. The Dick and Carey Model

The Dick and Carey model – also known as the instructional systems model – is an intricate, systematic approach that seeks to identify the specific components of learning and the relationship they have with one another. This model includes phases such as identifying instructional goals, analyzing tasks, designing teaching strategies, producing materials, and assessment.

3. The Four-Component Model (4C/ID)

The Four-Component Model (4C/ID) , developed by Jeroen J.G. Van Merriënboer, focuses on teaching complex tasks by integrating four main elements into the process: the learning of tasks; informational support; just-in-time support; and repeated practice. This model is especially useful for teaching complex, high-level skills, as it provides a structured yet flexible approach to instruction that accommodates deep learning, and the practical transfer of knowledge to real-life situations.

4. The ARCS Model

The ARCS model , developed by John Keller, concentrates on learner motivation and splits this into four key components: attention, relevance, confidence, and satisfaction . The model offers a range of specific strategies for capturing and maintaining students’ attention. It aims, for example, to: make content relevant to their needs and interests; build their confidence through productive, successful experiences; and ensure their satisfaction by allowing them to achieve learning objectives. ARCS is particularly helpful in situations where learning success involves student motivation.

PADDIE or PADDIE + M? The evolution of the ADDIE Model

As we’ve seen, the ADDIE model has been a fundamental tool in instructional design for several decades. However, as with any good methodology, it has continued to evolve over time in order to adapt to changing needs in training and technological advances within education. One of the most noteworthy changes that have resulted from this evolution is the transition to updated models such as PADDIE and PADDIE + M.

PADDIE: An added Planning stage

PADDIE is an evolution of the ADDIE model that includes an additional planning phase at the beginning of the process. This phase invites you to establish a solid understanding of the scope of your whole project before starting the instructional design process. This includes defining roles and responsibilities, creating detailed schedules, and ensuring that all the resources you need are available ahead of time. The addition of this phase confers a greater clarity on training objectives, and makes for more efficient management of resources as well as improved coordination.

PADDIE + M: An added Maintenance stage

The PADDIE + M model goes one step further again by adding a maintenance phase to the end of the cycle. This step focuses on continuously updating and improving the training course and its materials. Maintenance helps to ensure that content remains relevant and effective over time, adapting to new discoveries, sudden changes in the industry, technological advancements, and ongoing feedback from learners themselves.

The benefits of the maintenance phase include the ability to continuously update, and improve the training process, and the scope to adapt to changes in the educational, technological, or professional environment.

Conclusion: Is the ADDIE model right for you?

If you’re looking for a methodology that allows you to create effective and versatile training programs, then ADDIE may be the ideal choice for you. However, if you want to maximize the potential of this model, then it’s essential to have a robust platform that lets you management and deliver your training programs with agility. This is where isEazy LMS steps in to become your perfect partner. isEazy LMS combines the best of an LXP with the powerful features of an LMS to create a dynamic and intuitive platform – designed to simplify your training management process from start to finish.

With isEazy LMS, you can:

• Centralize your course management: organize and manage all your training materials from a single location.
• Personalize the learning experience: adapt content and learning pathways to the specific needs of your students.
• Automate processes: simplify administrative tasks so you can focus on what really matters – the quality of your training.
• Monitor and evaluate in real time: get accurate data on your students’ performance and keep up-to-date on the impact of your training programs.

Get going with isEazy LMS today! Request a demo and discover how you can transform your entire training process.

Frequently Asked Questions on the ADDIE Model

The ADDIE model was developed in the 1970s by Florida State University . It was initially intended to help the U.S. Army with efforts to standardize and systematize the process of creating military training materials.

The ADDIE model breaks down the creation of training into five phases: analysis; design; development; implementation; and evaluation . Each phase provides a structured approach to creating, implementing, and evaluating educational materials, utilizing a mixture of focused objectives and activities that raise the effectiveness of training programs.

The ADDIE model offers multiple benefits to instructional design, providing a clear and organized guide that structures the process, and focusing on the needs and objectives of both learners and companies. It also helps to steer projects towards the proposed goals, and works with an iterative approach, so training can be consistently improved through feedback and evaluation.

Some of the most common challenges faced when applying the ADDIE model include the time required to implement it (especially in large and complex projects), and t he need for trained staff and specific technological tools.

Yes. As a highly versatile model, it can be applied to a wide variety of training programs , including e-learning courses, interactive training modules, manuals, study guides, educational games and more. It’s just as adaptable to different learning environments, such as face-to-face, hybrid, and online, as well as to a diverse array of disciplines and industries.

CONTENT CREATED BY:

Elizabeth Aguiar Chacón

Content Marketing Specialist at isEazy

Featured articles

How to become a successful e-learning designer in 3 simple steps

The ADKAR Model: Discover how to drive change in your organization with this new trend

LMS vs. LXP: Which is the best option for your company? 

Essential technical skills to succeed in e-learning

Keep up to date with our news

MORE INFORMATION

Keep up to date with our news.

Cookies? They sound delicious

Services we would like to use.

Webinar en directo: Cómo utilizar la IA en tuherramienta de autor ¡Inscríbete!

Other featured articles

Design and share your own training content.

Bespoke content creation in record time.

Learning and management centred on the user experience.

A complete catalogue for upskilling your team.

The training app for frontline workers.

Gamification app for boosting the success of your training.

Sustainable app with a wide range of courses online.

Whitepapers

Case studies

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Research on comprehensive evaluation & development of aesthetic education based on PCA and CEM model

Roles Writing – original draft

Affiliation School of Art, Anhui University of Finance and Economics, Bengbu, China

Roles Supervision, Validation, Writing – review & editing

* E-mail: [email protected]

Roles Conceptualization, Data curation, Funding acquisition

Affiliation Graduate School, Hankuk University of Foreign Studies, Seoul, Korea

Roles Investigation, Methodology, Project administration

Affiliation School of Statistics and Applied Mathematics, Anhui University of Finance and Economics, Bengbu, China

Roles Project administration, Resources, Software

• Xin-Hong Xu, 
• Yu-Ting Niu, 
• Zhi-Min Li, 
• Yue-Yang Xu, 
• Ke-Wang Cao

• Published: August 9, 2024
• https://doi.org/10.1371/journal.pone.0308446
• Reader Comments

Aesthetic education, conveyed through public art courses, serves as a vital form of humanistic literacy education. It represents an effective approach to fostering innovative and creative thinking among college students. In order to effectively analyze the aesthetic education work of 46 universities, an aesthetic education index evaluation system is constructed, involving indicators including faculty strength, curriculum setting, teaching management, artistic practice, and teaching support. The secondary indicators corresponding to the five indicators are statistically analyzed, and a comprehensive evaluation analysis of the current development status of aesthetic education in 46 universities in Anhui Province is conducted by combining theoretical analysis with empirical analysis. Based on principal component analysis, an integrated evaluation model for the development of aesthetic education in universities in Anhui Province is further constructed. The model designed quantifies the influence weight of each aesthetic education index on the development of aesthetic education in Anhui Province, and forges a theoretical basis for determining the precursors of rapid development of aesthetic education in Anhui Province. Additionally, a novel approach is introduced to gauge the progression of aesthetic education within universities in Anhui Province, considering the dispersion of aesthetic education index data across the province. The comprehensive evaluation model for the development of aesthetic education in Anhui Province exhibits an overall declining trend. Hence, it is suggested to utilize the maximum value of the first derivative of the comprehensive evaluation model as an indicator of the imminent rapid development of aesthetic education in Anhui Province. On this basis, the probability equation of sustainable development of aesthetic education in Anhui Province is defined. Overall, the research results lay a theoretical foundation for the development of aesthetic education in Anhui Province.

Citation: Xu X-H, Niu Y-T, Li Z-M, Xu Y-Y, Cao K-W (2024) Research on comprehensive evaluation & development of aesthetic education based on PCA and CEM model. PLoS ONE 19(8): e0308446. https://doi.org/10.1371/journal.pone.0308446

Editor: Mc Rollyn Daquiado Vallespin, Far Eastern University - Manila, PHILIPPINES

Received: April 11, 2024; Accepted: July 23, 2024; Published: August 9, 2024

Copyright: © 2024 Xu et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All data are fully available without restriction.

Funding: The author(s) received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

1. Introduction

Aesthetic education, also referred to as aesthetic or art education, holds significant importance within higher education. It typically pertains to undergraduate programs lasting four or five years, excluding vocational and technical colleges, as well as graduate education for the time being. Based on the National General Art Education Curriculum Guidelines 2006 and the Opinions on Solidly Improving the Aesthetic Education Work in Colleges and Universities in the New Era 2020 [ 1 , 2 ], the results of aesthetic education in colleges and universities over the past 15 years indicate that on the one hand, aesthetic education in colleges and universities is still in the stage of theoretical exploration and practical education, while on the other, there is no feasible evaluation system available for the actual effect of aesthetic education. This results in an imbalance between excessive aesthetic education theory and the lack of evaluation system.

According to relevant documents such as China’s National Medium-to-Long-Term Education Reform and Development Plan 2010–2020 [ 3 ], Opinions on Comprehensively Strengthening and Improving Art Education in Schools [ 4 ], and Regulations on Art Education Work in Schools [ 5 ], this article examines 46 undergraduate universities in Anhui Province, adhering to principles of scientificity, comprehensiveness, and hierarchy. Meanwhile, it employs reverse reasoning and considers multiple observation points such as faculty strength, curriculum design, teaching management, artistic practice, and teaching support. Through the utilization of methods like principal component analysis, entropy weight method, and BP neural network method, the article offers a comprehensive evaluation of the current state of aesthetic education in universities in Anhui Province. Additionally, it highlights future challenges that need to be addressed.

2. Literature review

Aesthetic education is an education that cultivates people’s aesthetic perception, appreciation, creativity, and judgment. It holds irreplaceable significance in the higher education system, and presents long-term effectiveness, strong infectiousness, and transcends time, space, and ethnic and national aesthetic universality. Plato advocated for the cultivation of citizens in both body and mind as the ultimate goal of education, while Confucius emphasized the importance of poetry and music education as primary avenues for aesthetic education. Indeed, aesthetic education is an emotional education that concerns people mentality and personality development. When individuals are in crisis and pressure, aesthetic education can furnish them with a balanced, stable, and positive attitude to life [ 6 ]. Filipović and Vojvodić [ 7 ] believed that aesthetic education provides opportunities for students’ creative ability, creative communication ability, and aesthetic ability development, thereby affecting their personality emancipation and all-round development. In this way, art education makes people the most humanized and complete individual [ 8 ]. The implementation of aesthetic education in universities primarily involves utilizing public art courses as the foundation of art education. It centers on art works within the context of art history as the core content of the educational process. Art education refers to the education of music, dance, drama and visual arts disciplines [ 9 ]. Specifically, art encompasses various forms, including visual arts such as painting, drawing, sculpture, filmmaking, architecture, photography, and ceramic art. Additionally, it includes literary arts such as poetry, drama, prose, and novels, as well as performing arts like drama, dance, and music [ 10 ]. According to the National Common Propaganda on Public Art Education in Colleges and Universities in China , public art courses refer to those emphasizing aesthetics, art history and theory, art appreciation, and artistic practice represented by music, dance, painting, calligraphy, drama, film and television.

From the perspective of function and method of aesthetics, reconstructing STEM education by integrating art education is a way to implant art education [ 11 ]. STEAM combines science, technology, engineering, art, and mathematics education [ 12 ]. Meanwhile, art education can enhance students’ innovation ability and promote the cultivation of comprehensive innovative talents [ 13 ]. Scientific work requires creativity and critical thinking, while art and science mutually inspire each other in a subtle and covert way [ 14 ]. Wang and Zeng [ 15 ] believed that imaginative thinking ability is a kind of creative thinking ability, which is not only the core of art but also the key to scientific research. However, it is a challenge to scientifically evaluate the actual function of art in STEAM education from the student-centered perspective [ 16 ]. Teachers should have good adaptive teaching abilities to ensure that the aesthetic effect of public art courses is achieved [ 17 ]. In order to shift from the traditional aesthetic evaluation model centered on teachers, classrooms, and textbooks, a more student-centered approach can be adopted. This involves conducting aesthetic ability evaluations based on indicators such as artistic cultural knowledge and skills, artistic abilities, and artistic achievements. However, despite these efforts, the evaluation may still exhibit some bias towards a student-centered perspective [ 18 ]. In 2018, in order to construct a comprehensive evaluation system for higher education aesthetic education, Shandong Province established an evaluation index system for higher education aesthetic education work from five dimensions of curriculum design, teaching management, teacher team, artistic practice, and condition guarantee [ 19 ]. Only by adopting a more student-centered approach to aesthetic education evaluation can the aesthetic development of universities in the province truly benefit.

3. Comprehensive evaluation analysis

3.1 construction of the comprehensive evaluation index.

Aesthetic education as an educational phenomenon can be described, measured, calculated, and analyzed to help depict the quality of aesthetic education. Observing, describing, or researching the development of aesthetic education in various universities in Anhui Province aids in cultivating a reasoned comprehension of its status, distinctive features, and operational mechanisms. Comprehensive evaluation serves as a quantitative empirical method enabling the comparison of research objects through evaluation outcomes. Evaluating and comparing the progress of aesthetic education in universities across Anhui Province necessitates the establishment of a viable comprehensive evaluation index system, which should adhere to the following four principles:

The first principle is systematicity, where various indicators have logical relationships, reflecting the main characteristics and status of the overall system of aesthetic education from different aspects. The second is the principle of typicality, which emphasizes that indicators should possess certain representative characteristics to comprehensively reflect the current development status of aesthetic education. Selected indicators should have the capacity to guide the evaluation process effectively. The third principle involves independence, where the interaction of indicators within the system manifests as the characteristics of subsystems. The evaluation index system of aesthetic education includes independent subsystems reflecting teacher conditions, curriculum settings, teaching management, practical results, and teaching guarantees. The fourth principle encompasses comparability, operability, and quantifiability. The index system utilizes existing data and standards, ensuring consistency in the calculation methods and measurement of indicators. The indicators should be as concise as possible, with strong micro-level characteristics, easy to collect, and quantifiable.

Aesthetic education quality evaluation entails a comprehensive assessment and acknowledgment of the situation, processes, methods, objectives, and outcomes of education. This contributes significantly to the management, supervision, and enhancement of the quality and standard of education [ 20 ]. The evaluation of aesthetic education involves complexity and multidimensionality. A single indicator is far from sufficient to comprehensively and typically reflect its characteristics. Therefore, constructing a comprehensive evaluation index system requires decomposition, summarization, and refinement from multiple attributes, perspectives, and features [ 21 ]. Drawing on normative documents and prior scholarly research, an evaluation index system for the advancement of aesthetic education in universities within Anhui Province is hereby formulated, which comprises five sub-dimensions, i.e., teacher qualifications, curriculum design, teaching management, artistic practice, and teaching support, as delineated in Table 1 .

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0308446.t001

3.2 Data source and processing

To measure the development of aesthetic education in universities in Anhui Province, while ensuring the authenticity, validity, and accessibility of data, a variety of sources can be utilized. These may include annual reports on art development from each university, public documents from the Anhui Provincial Department of Education, official documents from each university’s website, and publications like the China Education Statistical Yearbook . These sources can be organized and compiled to form original cross-sectional data with a time variable set to 2020 for empirical analysis.

In a multi-indicator evaluation system, each indicator often varies in units and magnitudes due to their different nature. To address this, raw data undergo preprocessing, involving standardization and quantification of qualitative indicators. This ensures that all indicator values are on a uniform scale, facilitating comprehensive evaluation analysis.

This formula standardizes the raw data x nm , k for each indicator across all schools and years, ensuring comparability across different indicators.

3.3 Method description

In comprehensive evaluations, utilizing different methods provides analyses from various perspectives. To overcome the many differences in preference and adaptability inherent in single comprehensive evaluation methods, employing multiple evaluation methods is essential, followed by the integration of results [ 22 ]. This paper takes the 46 universities in Anhui Province as samples, constructs an indicator system, and summarizes and analyzes the development of aesthetic education in each university. Methods such as the entropy method, BP neural network method, and principal component analysis method are employed, followed by a combined evaluation approach.

The Back Propagation (BP) neural network comprehensive evaluation method offers a comprehensive and interactive approach. It effectively bypasses the inaccuracies associated with manually assigning weights and the complexities of solving correlation coefficients. Moreover, it enables comprehensive evaluation of instances with large quantities and numerous indicators [ 24 ]. The BP neural network is an artificial neural network composed of input, hidden, and output layers. The input layer nodes only receive signal inputs, while the output layer nodes perform linear weighting, and the hidden layer nodes primarily handle the most significant mathematical processing of information. During forward propagation, training samples traverse through the input layer, followed by the hidden layer, and finally to the output layer. If the output results fail to meet the expected output, the process will enter the backpropagation stage. Herein, the weights and thresholds of each neuron are continuously adjusted based on the set prediction error until approaching the expected output. The aesthetic education development in Anhui Province’s universities is assessed by inputting indicators from teacher qualifications, curriculum design, teaching management, artistic practice, and teaching support subsystems into Python. Leveraging the potent nonlinear mapping capability of BP neural networks, the data undergo iterative learning and optimization for a comprehensive evaluation.

Principal Component Analysis (PCA) is a multivariate statistical analysis method that reduces the dimensionality of data by linear transformation to select a smaller number of important variables. The basic idea is to recombine the original set of correlated indicators into a smaller set of uncorrelated composite indicators F m to replace the original indicators.

The above equation satisfies that F i and F j are uncorrelated, that is, cov( F i , F j ) = 0, and have nonzero covariances. F i is the linear combination of X 1 , X 2 , X 3 …. X p with the largest variance, that is, it is the combination of all linear combinations that are uncorrelated with each other and have the largest variance. It is constructed as the new variable indicator, namely the first, second,…, mth principal component of the original variable indicators. In practical applications, the specific steps of using Principal Component Analysis (PCA) are as follows: to standardize the original data; to calculate the covariance matrix; to find the eigenvalue λ i of R and the corresponding eigenvector a i of the orthogonalized unit; and to select principal components: through m in F 1 , F 2 , F 3 ….. F m is determined by the cumulative variance contribution rate G m .

Scores of aesthetic education in universities in Anhui Province are calculated, Principal Component Analysis is conducted using the Sklearn toolkit in Python, with the cumulative variance contribution rate set to be greater than 80%, and the scores for each university are finally computed.

3.4 Result interpretation

The entropy method, BP neural network and principal component analysis method are used to calculate the development scores of aesthetic education in 46 universities in Anhui Province. The weights of each index are shown in Table 2 :

https://doi.org/10.1371/journal.pone.0308446.t002

The weights of the indicators calculated by the three methods vary. The high rankings of both the number of teaching and research papers published by teachers and the number of art inheritance bases underscore their significance in supporting the development of aesthetic education. Research papers serve as vital data supporting the advancement of aesthetic education, while art inheritance bases enable schools, associations, and enterprises to leverage their strengths. They facilitate the discovery and cultivation of aesthetic education models and help standardize and guide aesthetic education behavior effectively. Hence, it is not difficult to understand that these two indicators occupy a large weight. In addition, significant weight is placed on the number of teachers participating in discipline competitions rated B or above, the ratio of traditional cultural associations to school associations, investment in aesthetic education funds, and the number of awards received in the previous National College Students Art Performance in Anhui Province.

After calculating the weights, the development scores of aesthetic education in 46 universities in Anhui Province are calculated according to three methods and evaluated jointly. Python software is used to draw the heat map, and the results are presented in visual form. The results are shown in Fig 1 .

https://doi.org/10.1371/journal.pone.0308446.g001

Fig 1 obviously reveals that darker colors correspond to higher scores, indicating a higher ranking in the development of aesthetic education. Observation over the last column of color blocks indicates that following the comprehensive evaluation, Anhui University, Anhui Normal University, Anqing Normal University, Chuzhou University, Anhui University of Finance and Economics, Fuyang Normal University, Anhui Academy of Fine Arts, Huangshan University, Anhui University of Engineering, and Bengbu University rank among the top ten. Reasons behind this include the fact that these universities offer five or more arts-related majors. Among them, Anhui Normal University, Anhui University, and Anqing Normal University offering over ten such majors. With sufficient funding and effective support for artistic venues, these universities prioritize professional faculty, thereby fostering comprehensive development in art courses, artistic heritage sites, and practical art endeavors. Additionally, institutions like Anhui Academy of Fine Arts specialize in cultivating artistic talents, positioning these universities at the forefront of aesthetic education development in the province.

4. Further analysis

Fig 2 illustrates a comparison of weighted indicators between research paper publications and the number of awards for artistic exhibitions across 46 universities in Anhui Province. From 2005 to 2022, in terms of the total number of awards at the national university art exhibitions organized by the Ministry of Education, universities with art schools, fine arts colleges, design colleges, and music colleges ranked prominently. A strong correlation between the quantity of research paper publications and the number of awards for artistic exhibitions is suggested by a calculated correlation coefficient of 0.899. Anhui Normal University, Anhui University, Suzhou University, Hefei Normal University, Anhui University of Finance and Economics, along with several other universities, exhibit a notable trend of both higher quantities of research paper publications related to aesthetic education and a greater number of awards for artistic exhibitions. This pattern implies the establishment of a systematic mechanism for nurturing talents, encompassing aspects such as faculty allocation, curriculum design, practical teaching methods, artistic practice, and participation in artistic exhibitions.

https://doi.org/10.1371/journal.pone.0308446.g002

In terms of overall statistical award results, participation in artistic exhibitions and subject competitions serves as a dual-purpose platform. It not only allows universities to showcase their achievements in aesthetic education but also serves as an observation point for assessing the degree to which universities prioritize efforts in aesthetic education. Through retrospective analysis, with a focus on professional settings, faculty allocation, and curriculum design for aesthetic education, it becomes evident that certain universities excel in constructing flagship or advantageous disciplines. However, this specialization may lead to fewer entries in exhibitions and competitions, consequently resulting in fewer awards. This situation could potentially relegate aesthetic education to a subsidiary or even marginal position within these universities.

An analysis is conducted on research papers regarding aesthetic education with significant weightage. Based on statistics from the China National Knowledge Infrastructure (CNKI) from 1990 to 2020, 624 research papers are examined. Python is employed for text segmentation and word frequency analysis to generate a word cloud, as shown in Fig 3 . The word cloud demonstrates "aesthetic education," "curriculum," and "art" as the three most frequently occurring terms, suggesting that art education, primarily focusing on public art courses, holds a dominant position. Among these, music and dance often occupy the forefront of aesthetic education. However, courses such as art appreciation, film and television appreciation, drama appreciation, calligraphy appreciation, and traditional Chinese opera appreciation are relatively marginalized. This is attributed to the prevalence of music and dance education at the primary and secondary school levels. Statistics also reveal that 26% of universities do not offer courses related to calligraphy, Chinese folk art, paper cutting, or intangible cultural heritage. Notably, Chinese calligraphy education appears to lack prominence, accounting for only 0.048% of research paper contributions. Furthermore, through a parallel comparison of public art courses, the structural imbalance in course offerings highlights the uneven distribution of such courses among the 46 universities.

https://doi.org/10.1371/journal.pone.0308446.g003

Differences in aesthetic education development in teaching management among the 46 universities in Anhui Province are further explored. Considering the small sample size and the teaching management indicators being categorical variables, a systematic cluster analysis method is adopted. The Ward method is used to calculate the distances between clusters using the squared Euclidean distance, and the number of clusters is determined based on the constructed pseudo-F statistic and pseudo-t 2 statistic. According to the statistics in Table 3 , nine clusters are finally obtained, as shown in Fig 4 .

https://doi.org/10.1371/journal.pone.0308446.g004

https://doi.org/10.1371/journal.pone.0308446.t003

The aesthetic education evaluation index system encompasses both basic statistics and variables. Hard indicators in teaching management include the presence of specialized public art course management departments, the number of teaching institutions, and the count of full-time staff in aesthetic education institutions or teaching departments. They have specific data requirements and are highly operational. The aesthetic management departments of the 46 universities vary widely, involving departments such as Art Education Centers, Cultural Arts and Quality Education Departments, Public Art Education Centers, Aesthetic Teaching Departments, Undergraduate Art Research Offices, Undergraduate Art Education Centers, Sports and Art Clubs, Public Art Education Clubs, Aesthetic Research Offices, etc. The diversity of management departments implies the flexibility of aesthetic education work, indicating an exploratory and practical stage in aesthetic education at universities. First, professional faculty and aesthetic education faculty may share resources, but this arrangement is uncertain and does not form a fixed teaching team. At the same time, soft indicators include the teaching syllabus of public art courses, the methods of public art course evaluation and assessment, public art course evaluations, awards, and teaching observation activities. Among these, cultural courses offered by engineering and medical universities occupy a relatively high proportion, but they are somewhat related to aesthetic education. The absence of standardized indicators among provincial universities limits the effectiveness of assessing procedural soft indicators, making it challenging to operationalize. The convergence of hard and soft indicators leads to a diverse range of teaching management approaches.

As shown in Fig 4 , a cluster analysis is conducted on the 46 universities in Anhui Province based on teaching management. The results indicate that Anqing Normal University, Chuzhou University, Anhui University of Finance and Economics, Tongling University, and Anhui Academy of Fine Arts are grouped into one cluster; Anhui University, Anhui Normal University, Anhui Agricultural University, Huangshan University, Bozhou University, Anhui Science and Technology University, and Hefei Normal University form another cluster; Bengbu Medical College, Wannan Medical College, Anhui Wenyi Information Engineering College, Fuyang Normal University School of Information Engineering, Anhui Medical University Clinical Medicine College, and Huaibei Normal University constitute a third cluster. The distribution of different types can be attributed to the setup of teaching management institutions. Specifically, 10 universities have dedicated institutions solely responsible for aesthetic education, supported by full-time staff. In contrast, 36 universities lack such dedicated staff, resulting in the delegation of aesthetic education responsibilities to various management departments. Regarding the establishment of art-related majors, six universities have more than 10 art-related majors, eight have between 7 and 9 art-related majors, 13 have between 4 and 6 art-related majors, 10 have between 1 and 3 art-related majors, and eight do not have any art-related majors. In terms of the student-teacher ratio for aesthetic education, according to the Guidelines for Public Art Courses in National Regular Higher Education Institutions , the number of teachers responsible for teaching public art courses in each school should account for 0.15% to 0.2% of the total number of students, with full-time teachers taking up 50% of the total number of art teachers. Based on the given ratio, certain medical and science/engineering universities have only one-third the necessary number of instructors for public art classes. Furthermore, universities with adequate faculty often fail to differentiate between art educators dedicated to specialized training and those teaching public art courses, thereby not offering a comprehensive array of public art courses. In terms of teaching management, responsibilities and personnel regarding faculty, courses, and practical activities should be further refined.

5. Aesthetic education comprehensive evaluation model

5.1 establishment of the model.

Herein, multiple indicators of aesthetic education development are selected to analyze and evaluate the status of aesthetic education in universities in Anhui Province. However, it remains challenging to quantitatively analyze aesthetic education development. The difficulty arises partly from some indicators being binary outcomes and the potential for the research conclusions to be limited to specific scenarios if only intuitive analysis of aesthetic education indicators is conducted. Therefore, five nonlinear indicators are hereby selected, which include the number of teacher publications on aesthetic education, the frequency of organizing school-level and above art activities, the number of awards in Anhui University Student Art Exhibition, the number of awards in provincial-level arts competitions, and the proportion of funding for cultural and folk heritage activities on campus-as the benchmark indicators for principal component analysis. A comprehensive evaluation model for aesthetic education development, considering data dispersion, is established. If the original variables are denoted as x 1 , x 2 ,⋅⋅⋅⋅, x n , and the new variables after principal component analysis are denoted as F 1 , F 2 ,⋅⋅⋅⋅, F m , where m < n, the new variables are linear combinations of the original variables. The new variable coordinate system is derived by translating and rotating the original one. The resulting space, comprising the new variables, is termed the m-dimensional principal hyperplane. On this plane, the first principal component F 1 corresponds to the direction of the largest data variation (contribution rate e1), while for F, there are e 2 ≥ e 3 ≥ ⋅⋅⋅≥ e m in sequence. As a result, the new variables effectively encapsulate the essence of the original data, while the m-dimensional hyperplane holds the utmost information content. Despite the possible presence of a slight loss of data information, this method effectively captures the primary contradictions, extracting the majority of variance from the original variables. By reducing the number of variables and focusing on the essential information, it facilitates problem analysis and processing, streamlining the overall approach.

Through principal component analysis, some of the original data information with minor contributions to the final results can be eliminated, while the main characteristic information of aesthetic education development indicators is retained. This is conducive to the analysis and processing of aesthetic education development in Anhui Province. The calculation process of the comprehensive evaluation model for aesthetic education is illustrated in Fig 5 .

https://doi.org/10.1371/journal.pone.0308446.g005

5.2 Precursors of rapid development in art education

The weight analysis in Principal Component Analysis (PCA) establishes a theoretical foundation for pinpointing sensitive indicators that mirror the advancement of art education in Anhui Province. Through PCA, a comprehensive evaluation model of art education development in Anhui Province is constructed, which incorporates multiple parameters such as the number of published papers on art education by teachers, the frequency of organizing arts events at the school and higher levels, the number of awards won by university students in the Anhui division of art exhibitions, the number of awards won in arts competitions at the provincial level and above, and the proportion of funding allocated to cultural heritage and folk art projects entering campuses. This model streamlines a comprehensive analysis of art education development in Anhui Province. Its essence lies in summing the product of each indicator and its corresponding weight across different dimensions of art education in higher education institutions within the province. A thorough investigation into the precursors of rapid development in art education aids in understanding the influence of various factors such as societal, policy, and technological factors on art education. It provides a basis for devising more effective strategies for art education development and lays the theoretical groundwork for the sustainable development of art education in Anhui Province.

Fig 6 depicts the evolution curve of the comprehensive evaluation model of art education development in Anhui Province as a function of funding input. As illustrated in the figure, the curve of the comprehensive evaluation model for art education development in Anhui Province exhibits an overall decreasing trend, with values ranging from -0.8 to 0.2. Despite the variations in numerical values and diverse changing trends among different art education indicators, indicating a certain degree of dispersion, this study takes into account the dispersion of data across these indicators. Polynomial fitting is applied to the curve of the comprehensive evaluation model for art education development in Anhui Province, using logarithmic, exponential, and polynomial functions. It is found that the correlation coefficient of the fitting curve reaches its highest value of 0.999 when a third-degree term is used, indicating a near-complete overlap between the fitting curve and the comprehensive evaluation model curve. Subsequently, the equation of the fitting curve is differentiated once and twice. The first derivative’s significance lies in indicating the rate of change of the curve in the comprehensive evaluation model for art education development in Anhui Province. It indirectly reflects the model’s sensitivity to changes in art education funding. Extreme values of the first derivative signify transitions in sensitivity states. The derivative curve of the comprehensive evaluation model for art education development in Anhui Province is shown in Fig 6 , which exhibits an overall increasing followed by decreasing trend. As art education funding increases, the first derivative peaks, with the second derivative reaching 0. Herein, the authors pinpoint this maximum of the first derivative as the precursor to rapid development in art education. At this juncture, funding input for art education in Anhui Province stands at 1,331,000 RMB, presenting a corresponding comprehensive evaluation model value of -0.502.

https://doi.org/10.1371/journal.pone.0308446.g006

5.3 Sustainable development probability of aesthetic education

Art education holds considerable significance in the sustainable development of society, emphasizing the cultivation of students’ creative thinking and innovation skills, both essential components for the enduring development of society. Creativity is a key factor driving social progress and problem-solving. Art education stimulates individuals’ independent thinking, fostering creativity and providing ongoing momentum for societal innovation. It helps to inherit and promote local culture while also facilitating communication and integration between different cultures. Art facilitates a deeper understanding and respect for multiculturalism, fostering a more harmonious and inclusive society. Besides, art education emphasizes cultivating students’ aesthetic tastes and cultural literacy, enriching individuals in the arts and enhancing their overall comprehensive qualities. Additionally, participation in artistic activities cultivates skills in interpersonal communication, cooperation, and collaboration, which are also crucial for societal sustainable development. The cultural and creative industries are integral components of modern society’s economy. Also, art education nurtures creative talents like artists, designers, and cultural professionals, fostering the development of the creative industry and establishing a cultural and economic foundation for societal sustainability. Moreover, it cultivates environmental awareness by drawing attention to environmental issues. At the same time, artworks serve as powerful tools for conveying environmental concepts, arousing concern through visual and aesthetic means, and advocating for sustainable environmental development.

When t = T, the art education funding input in Anhui Province reaches its peak. Hence, the probability value of sustainable development of art education at the peak state is 1. By computing the probability density curve of sustainable development of art education in Anhui Province based on Eq (16) , Fig 7 illustrates the evolutionary trend of the probability density of sustainable development of art education in Anhui Province. As shown in the figure, the probability density curve of sustainable development of art education in Anhui Province exhibits an overall trend of accelerating increase. The probability value of sustainable development corresponding to the precursor point of rapid development of art education in Anhui Province is 0.388. The probability values of sustainable development of art education corresponding to art education funding of 1.85 million yuan and 1.95 million yuan are 0.8 and 0.9, respectively. Examining the probability curve of sustainable art education development in Anhui Province provides access to probability values for sustainable development across different levels of art education funding.

https://doi.org/10.1371/journal.pone.0308446.g007

6. Discussions and recommendations

In Anhui Province, 10 colleges and universities offer one to three art majors, while 8 lack any art programmes. Some medical and scientific institutions have a mere one-third of the public art faculty compared to the program’s needs. Additionally, colleges with adequate faculty do not segregate those focused on professional art education from public art teachers, resulting in an incomplete public art curriculum. The authors believe that a sound teaching management organisation should be established. On the basis of the 10 colleges and universities that already have had an organisation specifically responsible for aesthetic education, it is recommended that a specialised aesthetic education management organisation with full-time staff also be established in the remaining colleges and universities. This ensures a specialised team is dedicated to overseeing and advancing aesthetic education. Secondly, the setting of art majors should be optimised. For the 8 colleges and universities without art majors, more relevant majors should be gradually set up to enrich the disciplines. Meanwhile, institutions with a limited number of art majors should be motivated to expand their offerings and evolve towards a more diverse and holistic approach. This strategy aims to accommodate the varied demands of society for artistic talent and to bolster the overall competitiveness of these colleges and universities. Colleges and universities, particularly those focused on medical and technical fields, should bolster their recruitment efforts for public art programs. This is to ensure that the faculty count meets the necessary standards, thereby enhancing the quality and breadth of aesthetic education. In addition, it is recommended that full-time art teachers and public art course teachers be divided to undertake the teaching tasks of professional education and public art courses, respectively. This ensures that the courses are fully opened. In terms of teacher management, the teacher training and assessment mechanism should be strengthened, and art teachers should be organised to participate in training and exchange activities on a regular basis, thereby improving their teaching ability and professionalism. At the same time, a scientific teacher appraisal system can be established to evaluate the teaching effectiveness and performance, so as to motivate the teachers for consistent self-improvement and optimisation of the teaching quality. In terms of curriculum, the public art curriculum system should be enriched and improved. In alignment with the distinct profiles of colleges and universities and the varied interests of their students, a diverse curriculum of art courses, spanning music, visual arts, dance, theater, and other disciplines, should be established. This curriculum is designed to cater to the broad artistic learning needs of students, enrich their practical experience, and enhance their practical skills and overall quality through a combination of campus-based and extramural internships and art practice activities.

Finally, delineating clear responsibilities and assigning specific roles is essential. Create a personalized responsibility system to ensure that the task of aesthetic education work is decomposed into specific posts and individuals, so that every task has a responsible person in charge and is subject to proper supervision. By clearly defining roles and enforcing accountability, work efficiency is enhanced, ensuring the seamless execution of aesthetic education initiatives. For example, by adding a number of art majors and actively bringing in high-calibre teachers, Zhejiang University has rapidly formed a competitive cluster of art disciplines and increased the university’s influence in the field of art education. Meanwhile, the Chinese University of Hong Kong has significantly enhanced students’ artistic qualities and practical abilities through the provision of diversified art courses and the organisation of rich art practice activities. However, the above proposal also faces challenges on various fronts, including problems in funding, resources, time and management co-ordination. Specifically, the introduction of additional specialisations necessitates considerable financial and resource investment, potentially burdening HEIs with scarce resources. Recruiting and training adequate faculty, notably in medical and polytechnic institutions, is a protracted process. Crafting and launching a varied curriculum demands significant time and resources, potentially overwhelming current educational capacities. Moreover, organizing and overseeing practical sessions involves multi-party resource coordination, presenting logistical challenges. In order to achieve an overall improvement in aesthetic education, successful implementation of these recommendations requires detailed planning and step-by-step progress by the universities according to their own specific conditions, balancing short-term goals with long-term development.

The comprehensive evaluation model established in this paper serves as a novel method for quantitatively analysing the development degree of aesthetic education. Its theoretical basis relies on the discrete characteristics of aesthetic education metrics, utilizing mathematical curve-fitting techniques, along with the application of primary and secondary derivatives for analysis. The model can effectively analyse the relationship between the input of aesthetic education and the degree of development. The primary derivative reflects the sensitivity of the comprehensive evaluation model to changes in the funding of aesthetic education, while the point of extreme value indicates a shift in the state of sensitivity. The extreme value point of the primary derivative is considered the precursor point of rapid development of aesthetic education, similar to the inflection point in the theory of marginal effect, revealing the rapid growth of aesthetic education input funding upon reaching a certain degree. In order to measure the sustainable development characteristics of aesthetic education, the authors have integrated the concept of probability value. They have delineated the probability of sustainable development in aesthetic education, grounded in a comprehensive evaluation model.

In terms of practical application, the model proposed in this paper can assist policy makers and educational administrators in identifying the key points of investment in aesthetic education, optimising the funding allocation strategy, and ensuring the effective use of resources. Analysing the derivative curves enables educational administrators to dynamically evaluate the responsiveness of aesthetic education development to inputs and the effectiveness of those inputs. This provides data support for the continuous improvement of the effectiveness of the aesthetic education teaching and learning. By identifying the precursor points of rapid development, the model can facilitate colleges and government departments to formulate long-term planning and predict the future trend of aesthetic education development. The points at which both the first and second derivatives’ extreme values equal zero signify a pivotal shift in the sensitivity of funding inputs to the advancement of aesthetic education. These points corroborate the hypothesis that the field is poised for rapid development. In terms of model validation, in future studies, the authors will apply the model to the data on aesthetic education in other provinces to validate the generalisability of the model. In addition, the accuracy of the model predictions will be verified by tracking the actual data on the development of aesthetic education in Anhui Province over time. This paper constructs a comprehensive evaluation model of aesthetic education in universities in Anhui Province based on principal component analysis. Besides, it identifies the precursor points for the rapid development of aesthetic education, and defines the probability value of the sustainable development of aesthetic education in Anhui Province. The paper’s model excels by accommodating the discrete nature of data, making it versatile for comprehensive aesthetic education evaluations across diverse datasets. Yet, its efficacy is critically dependent on the quality and completeness of aesthetic education indicator data.

In addition, the model mainly analyses the relationship between the investment in aesthetic education and the development degree, failing to comprehensively consider other factors affecting the development of aesthetic education, such as cultural background, policy environment, and students’ interests. As artificial intelligence advances rapidly, the authors intend to leverage AI technology in their forthcoming research for mining and analysing aesthetic education big data. This will encompass a broader spectrum of data related to aesthetic education, including student feedback, assessments of teaching quality, and the social impact of these educational efforts. More comprehensive analytical support will be provided. Furthermore, the authors aim to employ deep learning models to assess the collective impact of various factors on the progression of aesthetic education. These models will predict how the field may evolve under diverse conditions, thereby enhancing the precision of decision-making support. On this basis, natural language processing technology will also be utilized to analyse literature, policy documents and expert opinions related to aesthetic education. Valuable information can be extracted for model optimisation and strategy formulation, while an intelligent recommendation system can be developed to recommend optimal aesthetic education development strategies and resource allocation plans according to the specific conditions of universities. In this case, the efficiency and effectiveness of aesthetic education work can be comprehensively enhanced. The paper’s research contribution is exemplified through the case study of Anhui Province, where it introduces a comprehensive evaluation model for aesthetic education. Besides, it defines the probability value of sustainable development of aesthetic education in Anhui Province. The paper’s model stands out for its ability to handle the discrete nature of data, making it suitable for conducting comprehensive evaluations of aesthetic education across a range of data scenarios. However, the size of the data volume may affect the analysis scope of the model, yet this limitation does not impact data accuracy or the model’s predictive trends and policy suggestions for aesthetic education. The authors plan to collect a larger volume of data in future studies to enhance the comprehensive evaluation model’s analysis capabilities.

7. Conclusion

Following a thorough evaluation of art education development across 46 universities in Anhui Province utilizing a construction of 25 indicators, it becomes evident that there are variations in the development of art education among these universities. For instance, Anhui University and Anhui Normal University, representing comprehensive and normal universities respectively, rank higher in terms of overall strength and university rankings. However, universities such as the University of Science and Technology of China, Hefei University of Technology, and Anhui Medical University, despite their high overall strength, do not necessarily lead in art education. Despite advancements in higher education, art education continues to be a weak aspect within the system. However, universities such as Chuzhou University and Huangshan University, while ranking average in overall university standings, demonstrate relatively advanced development in art education.

In summary, several observations arise: Firstly, universities housing art colleges boast advantages in faculty expertise. However, there is still a need to differentiate between specialized courses for art students and those for the general public. Secondly, universities without art colleges generally offer public arts courses in literature and culture. Thirdly, while some universities offer distinctive interdisciplinary courses such as Contemporary Science and Technology Arts (USTC), Medical Humanities Film (BBMC), High-tech Ceramic Materials (AHUT), and Architectural Aesthetics (AHJZU), there is still a relative scarcity of courses focusing on interdisciplinary studies and fostering students’ innovative thinking.

According to the undergraduate talent training regulations of the 46 universities, students must complete 2 credits of public arts courses to graduate during their school years. However, most universities have not fully offered public arts courses, leading to varying degrees of imbalance between supply and demand. Consequently, students may not necessarily fulfill their 2-credit art education based on their interests, hobbies, and specialties. Art education may even become utilitarian education driven by credit orientation, neglecting the importance of aesthetic consciousness, personality development, and innovative thinking, making it necessarily important to strengthen faculty construction and curriculum system development for public arts courses.

Among the forty-six universities in the province, the total number of student art groups and art-related clubs ranges from 5 to 30. However, a higher quantity of literary and art clubs does not necessarily indicate more artistic practice activities, as some clubs may be dormant. Indeed, art education extends beyond formal coursework in public arts courses; it encompasses practical experiences like visiting art galleries, museums, and attending art exhibitions. Additionally, various professional education programs also incorporate elements of art education. Therefore, the construction of a comprehensive evaluation index system for faculty strength, curriculum design, teaching management, artistic practice, and teaching support matters considerably for improving the quality of higher education. The next step in enhancing art education involves focusing on faculty quality, uniqueness, practicality, and innovation in the reform of public arts course teaching.

Herein, a comprehensive evaluation model for the development of art education in Anhui Province is constructed based on principal component analysis. This model quantifies the impact weights of various art education indicators on the development of art education in Anhui Province, and also provides a theoretical basis for identifying the precursors of rapid development of art education in Anhui Province. Furthermore, a new method for determining the development of art education in Anhui Province is also proposed, which takes into account the discreteness of art education indicator data in Anhui Province. The comprehensive evaluation model curve for art education development in Anhui Province exhibits an overall declining trend. Building upon this observation, the authors suggest utilizing the maximum value of the first derivative of the comprehensive evaluation model as the precursor point for rapid art education development in Anhui Province. This very approach leads to the definition of a probability equation for the sustainable development of art education in the province, forging a theoretical foundation for its future development.

• 1. MOE of PRC. The Guidance Program of Public Art Curriculum of National Higher Education Institutions, Retrieved 24 May, 2022, from http://www.moe.gov.cn//srcsite/A17/moe_794/moe_624/200603/t20060308_80347.html .
• 2. General Office of the CPC Central Committee, General Office of the State Council of PRC. Opinions on Effectively Strengthening Aesthetic Education in Higher Education in the New Era, Retrieved 20 March, 2022, from http://www.gov.cn/gongbao//content/2020/content_5554511.htm .
• 3. MOE of PRC. Outline of National Medium and Long-term Education Reform and Development Plan (2010), Retrieved 17 June, from http://www.moe.gov.cn/srcsite/A17/moe_794/moe_624/201904/t20190411_377523.html .
• 4. MOE of PRC. Opinions on Comprehensively Strengthening and Improving Aesthetic Education in Schools (2015), Retrieved 8 August, 2022, from http://www.moe.gov.cn/s78/A01/s4561/jgfwzx_zcwj/202010/t20201019_495584.html .
• 5. MOE of PRC. School Art Education Work Regulations (2002), Retrieved 25 October, 2022, from http://www.moe.gov.cn/srcsite/A02/s5911/moe_621/200207/t20020725_81854.html .
• View Article
• Google Scholar
• 8. Richard PJ, Thelma CA (2012). The Art of Being Human, tenth edition, New York: Pearson Education, inc.
• 11. Howes A, Kaneva D, Swanson S, Williams J (2013). Re-envisioning STEM Education: Curriculum, Assessment and Integrated, Interdisciplinary Studies. The University of Manchester, pp.1–55.
• 15. Wang M, Zeng FR (2017). The Modern Construction of a Large Aesthetic Education System in Colleges and Universities. Chinese Higher Education, no.7, (General Serial 583), pp.7-10.
• 18. Shi CX (2020). Research on the Evaluation System and Promotion Strategies of College Students’ Artistic Quality under the Background of Aesthetic Education. China University Teaching, no.6 (General Serial 358), pp.60-63.
• 19. Shandong Provincial Department of Education. Evaluation Index System of Aesthetic Education Work in Higher Education Institutions in Shandong Province. https://edu.shandong.gov.cn . Accessed 24 March, 2022.
• 21. Yin YX (2020). Research on the Current Situation and Countermeasures of Aesthetic Education in Science and Engineering Colleges and Universities. Nanjing University of Technology. https://doi.org/10.27238/d.cnki.gnjhu.2020.000064
• 24. Wang XH, Li TJ, Ding LL (2010). Evaluation Theory and Techniques for Complex Large Systems. Jinan: Shandong University Press, pp. 80–89.
• 25. Yuan YC (2017) Evolution Law of Precursor Information of Sudden Water Influx in Tunnels and Fusion Warning Method and Engineering Application [D]. Jinan: Shandong University.
• 28. Sheng Z, Xie SQ, Pan CY (2008). Probability Theory and Mathematical Statistics[M]. Beijing: Higher Education Press.

Research on new power system research and development enterprise contribution evaluation model based on FUCOM-BWM and improved TOPSIS

New citation alert added.

This alert has been successfully added and will be sent to:

You will be notified whenever a record that you have chosen has been cited.

To manage your alert preferences, click on the button below.

New Citation Alert!

Please log in to your account

Information & Contributors

Bibliometrics & citations, index terms.

Applied computing

Operations research

Decision analysis

Multi-criterion optimization and decision-making

Recommendations

Research on enterprise informatization evaluation system.

In this paper, from a strategic point of view of enterprise informatization, EICMM (Enterprise Informatization Capacity Maturity Model) system and the model system of its impact factors are put forward, the multi-index enterprise informatization ...

Research and Practical Issues of Enterprise Information Systems II Volume 2: Ifip Tc 8 Wg 8.9 International Conference on Research and Practical

Research on enterprise competitiveness evaluation based on topsis and wavelet neutral network.

The competitiveness evaluation of enterprises has important significance for the survival and development of enterprises, and also has important reference value for enterprises decision-making. In this paper, Evaluation model of enterprise ...

Information

Published in.

Association for Computing Machinery

New York, NY, United States

Publication History

Permissions, check for updates.

• Research-article
• Refereed limited

Contributors

Other metrics, bibliometrics, article metrics.

• 0 Total Citations
• 0 Total Downloads
• Downloads (Last 12 months) 0
• Downloads (Last 6 weeks) 0

View Options

Login options.

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

View options.

View or Download as a PDF file.

View online with eReader .

HTML Format

View this article in HTML Format.

Share this Publication link

Copying failed.

Share on social media

A Comparative Study of the ADDIE Instructional Design Model in Distance Education

• August 2022
• Information 13(9):402
• This person is not on ResearchGate, or hasn't claimed this research yet.

• International Hellenic University

• University of Ioannina

Abstract and Figures

Discover the world's research

• 25+ million members
• 160+ million publication pages
• 2.3+ billion citations

• Educ Inform Tech

• Shujuan Zhang

• Cantika Putri

• Ferric Limano

• Dety Amelia Karlina

• Andromachi Filippidi
• Mega Novela Ulina Manurung

• Xufeng Huang
• Muhammad Haddad Richard
• Anisatu Thoyyibah

• Sylvester Arnab

• Diana Andone

• Alessandro Liberati
• Douglas G. Altman

• David Moher

• Shih-Jou Yu
• Yu-Ling Hsueh

• J ACAD LIBR
• Emily Johnson-Barlow
• Carl A. Lehnen
• Hanwen Dong
• Recruit researchers
• Join for free
• Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up

This paper is in the following e-collection/theme issue:

Published on 8.8.2024 in Vol 26 (2024)

A 3D and Explainable Artificial Intelligence Model for Evaluation of Chronic Otitis Media Based on Temporal Bone Computed Tomography: Model Development, Validation, and Clinical Application

Authors of this article:

Original Paper

• Binjun Chen 1, 2 * , MD   ; 
• Yike Li 3 * , MD, PhD   ; 
• Yu Sun 4 * , MD, PhD   ; 
• Haojie Sun 1, 2 , MD   ; 
• Yanmei Wang 1, 2 , MD, PhD   ; 
• Jihan Lyu 1, 2 , MD   ; 
• Jiajie Guo 5 , PhD   ; 
• Shunxing Bao 6 , PhD   ; 
• Yushu Cheng 7 , MD   ; 
• Xun Niu 4 , MD   ; 
• Lian Yang 8 , MD   ; 
• Jianghong Xu 1, 2 , MD, PhD   ; 
• Juanmei Yang 1, 2 , MD, PhD   ; 
• Yibo Huang 1, 2 , MD, PhD   ; 
• Fanglu Chi 1, 2 , MD, PhD   ; 
• Bo Liang 8 * , MD   ; 
• Dongdong Ren 1, 2 * , MD, PhD  

1 ENT Institute and Department of Otorhinolaryngology, Eye & ENT Hospital, Fudan University, Shanghai, China

2 NHC Key Laboratory of Hearing Medicine Research, Eye & ENT Hospital, Fudan University, Shanghai, China

3 Department of Otolaryngology—Head and Neck Surgery, Vanderbilt University Medical Center, Nashville, TN, United States

4 Department of Otorhinolargnology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China

5 State Key Laboratory of Digital Manufacturing Equipment and Technology, School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, China

6 Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, United States

7 Department of Radiology, Eye & ENT Hospital, Fudan University, Shanghai, China

8 Department of Radiology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China

*these authors contributed equally

Corresponding Author:

Yike Li, MD, PhD

Department of Otolaryngology—Head and Neck Surgery

Vanderbilt University Medical Center

1215 Medical Center Drive

Nashville, TN, 37232

United States

Phone: 1 6153438146

Email: [email protected]

Background: Temporal bone computed tomography (CT) helps diagnose chronic otitis media (COM). However, its interpretation requires training and expertise. Artificial intelligence (AI) can help clinicians evaluate COM through CT scans, but existing models lack transparency and may not fully leverage multidimensional diagnostic information.

Objective: We aimed to develop an explainable AI system based on 3D convolutional neural networks (CNNs) for automatic CT-based evaluation of COM.

Methods: Temporal bone CT scans were retrospectively obtained from patients operated for COM between December 2015 and July 2021 at 2 independent institutes. A region of interest encompassing the middle ear was automatically segmented, and 3D CNNs were subsequently trained to identify pathological ears and cholesteatoma. An ablation study was performed to refine model architecture. Benchmark tests were conducted against a baseline 2D model and 7 clinical experts. Model performance was measured through cross-validation and external validation. Heat maps, generated using Gradient-Weighted Class Activation Mapping, were used to highlight critical decision-making regions. Finally, the AI system was assessed with a prospective cohort to aid clinicians in preoperative COM assessment.

Results: Internal and external data sets contained 1661 and 108 patients (3153 and 211 eligible ears), respectively. The 3D model exhibited decent performance with mean areas under the receiver operating characteristic curves of 0.96 (SD 0.01) and 0.93 (SD 0.01), and mean accuracies of 0.878 (SD 0.017) and 0.843 (SD 0.015), respectively, for detecting pathological ears on the 2 data sets. Similar outcomes were observed for cholesteatoma identification (mean area under the receiver operating characteristic curve 0.85, SD 0.03 and 0.83, SD 0.05; mean accuracies 0.783, SD 0.04 and 0.813, SD 0.033, respectively). The proposed 3D model achieved a commendable balance between performance and network size relative to alternative models. It significantly outperformed the 2D approach in detecting COM ( P ≤.05) and exhibited a substantial gain in identifying cholesteatoma ( P <.001). The model also demonstrated superior diagnostic capabilities over resident fellows and the attending otologist ( P <.05), rivaling all senior clinicians in both tasks. The generated heat maps properly highlighted the middle ear and mastoid regions, aligning with human knowledge in interpreting temporal bone CT. The resulting AI system achieved an accuracy of 81.8% in generating preoperative diagnoses for 121 patients and contributed to clinical decision-making in 90.1% cases.

Conclusions: We present a 3D CNN model trained to detect pathological changes and identify cholesteatoma via temporal bone CT scans. In both tasks, this model significantly outperforms the baseline 2D approach, achieving levels comparable with or surpassing those of human experts. The model also exhibits decent generalizability and enhanced comprehensibility. This AI system facilitates automatic COM assessment and shows promising viability in real-world clinical settings. These findings underscore AI’s potential as a valuable aid for clinicians in COM evaluation.

Trial Registration: Chinese Clinical Trial Registry ChiCTR2000036300; https://www.chictr.org.cn/showprojEN.html?proj=58685

Introduction

Chronic otitis media (COM) represents a recurrent inflammatory condition inside the tympanic cavity [ 1 ]. COM encompasses various forms, including chronic suppurative otitis media (CSOM) and cholesteatoma, each with unique histological characteristics. CSOM involves the accumulation and discharge of purulent fluid, affecting an estimated 330 million people worldwide, with approximately half experiencing hearing loss [ 2 ]. Cholesteatoma is characterized by the buildup of keratinized squamous epithelium, which has the potential to erode auditory structures and exhibits a notable tendency for relapse. Accurate identification and differentiation of COM types are crucial for effective disease management and surgical planning [ 3 ]. Mastoidectomy, which involves the removal of part of the temporal bone, is the conventional surgical approach for COM. However, less invasive techniques such as endoscopic tympanoplasty are gaining favor for treating CSOM and other noncholesteatoma conditions due to their potential for reduced structural damage and faster recovery [ 4 - 9 ].

Temporal bone computed tomography (CT) is vital for assessing COM and aiding in surgical planning, especially when initial otoscopic examinations have restricted views and yield inconclusive findings [ 10 ]. Offering a cost-effective alternative to magnetic resonance imaging (MRI), CT is instrumental in distinguishing cholesteatoma from CSOM by detecting osseous erosion in the tympanum. Although studies have shown that clinicians are capable of diagnosing COM based on CT alone [ 11 - 17 ], distinguishing between COM subtypes poses greater challenges to the human eye. Moreover, interpreting temporal bone CT scans requires specialized training and experience, which may not be universally available across otolaryngologists.

Artificial intelligence (AI) is making remarkable advancements in health care. Deep learning (DL) models, particularly convolutional neural networks (CNNs), have demonstrated enhanced efficiency and reduced errors in disease diagnoses and prediction of clinical outcomes [ 18 - 21 ]. While a few recent papers have reported CNN models in evaluating COM with accuracy scores ranging from 0.77 to 0.85, these studies primarily relied on otoscopic or single-layer CT scans [ 22 , 23 ]. These 2D representations may not be optimal for revealing pathological changes in concealed or peripheral anatomical structures, such as the attic space and the mastoid air cells. In addition, the inherent “black box” nature of DL models, where decision-making strategies are challenging to understand, has been a common criticism [ 24 , 25 ]. This lack of comprehensibility hinders the widespread adoption of AI models in clinical practice.

In light of these challenges, this study aimed to create an explainable, 3D CNN model for the automatic interpretation of temporal bone CT scans. The model was designed to pinpoint the region of interest (ROI) and identify pathological and cholesteatomatous conditions in a 3D fashion. Comprehensive benchmarks against baseline methods and human experts on distinct data sets were conducted to demonstrate the robustness and generalizability of this model. In addition, heat map generation was used to highlight potential pathological changes in CT scans and elucidate the model’s rationale for making predictions. These features were integrated into an AI system for the automatic, end-to-end evaluation of COM, which was subsequently assessed in clinical settings. The overarching goal of this system is to support clinicians in making informed decisions for common otologic conditions, thereby enhancing efficiency, reliability, and transparency.

Ethical Considerations

This study was conducted in accordance with the principles of the Declaration of Helsinki. Ethical approval was granted by the institutional review boards at Vanderbilt University Medical Center (191804) and the Eye, Ear, Nose and Throat (EENT) Hospital of Fudan University (2019076). Informed consent was waived as all data were de-identified. The observational study, which aimed to assess the model’s viability in aiding preoperative assessment, was registered with the Chinese Clinical Trial Register (ChiCTR: 2000036300). No compensation was provided to any study participants.

Participants

Data were retrospectively obtained from patients admitted for middle ear surgeries from December 2015 to July 2021 at EENT Hospital. Patients diagnosed with acute otitis media, any inner or external ear diseases, or those with missing temporal bone CT scan were excluded, resulting in 1661 patients eligible for model development. An extra data set containing 108 patients with COM was collected from Wuhan Union (WU) Hospital for external validation ( Figure 1 ).

Temporal Bone CT Scans

As part of the routine preoperative assessment, each patient underwent at least 1 temporal bone CT, conducted from the lower margin of the external auditory meatus to the top margin of the petrous bone using a SOMATOM Sensation 10 CT scanner (Siemens Inc) at the EENT Hospital. The scanning parameters were as follows: matrix (512 × 512), field of view (220 mm × 220 mm), tube voltage (140 kV), tube current (100 mAs), section thickness (0.6-0.75 mm), window width (4000 HU), and window level (700 HU). CT scans from the WU Hospital were obtained using a SOMATOM Plus 4 model (Siemens Inc) with different settings for field of view (100 mm), voltage (120 kV), and thickness (0.75 mm). All images were saved in the DICOM format.

Label Assignment

All eligible ears were treated as independent cases and assigned ground truth labels based on their diagnoses ( Table 1 ). Each label was verified according to intraoperative findings and pathology reports for operated ears and using a combination of history, ear examination, audiogram results, and imaging findings for unoperated ears. In cases of unoperated ears, a “normal” label was assigned when there was an absence of ear symptoms, hearing loss, or signs of inflammation. A diagnosis of CSOM was assigned when chronic purulent discharge, conductive hearing loss, and the presence of a perforated tympanic membrane or soft tissue shadow in the tympanic cavity were observed. Cholesteatoma was considered if keratin debris was identified, or if there were signs of osseous damage along with retraction or perforation of the pars flaccida [ 22 ]. Two otolaryngology residents with full access to patients’ medical records independently reviewed these labels as unblinded annotators. Any discrepancies were addressed with senior specialists until a consensus was reached. All data were deidentified and stored on password-protected computers.

a EENT: Eye, Ear, Nose, and Throat Hospital of Fudan University.

b WU: Wuhan Union Hospital.

c CSOM: chronic suppurative otitis media.

d OME: otitis media with effusion.

Model Architecture

The framework consists of 2 functionally distinct units: a region proposal network for 3D segmentation of ROI, and a classification network for generating predictions. Both networks are established based on CNN models.

Region Proposal Network

This network is designed to extract the middle ear on each side from a full set of temporal bone CT scan ( Figure 2 A). It contains a YOLO (You Only Look Once; v5) model that is trained to detect and locate 2 auditory structures, including the internal auditory canal and the horizontal semicircular canal, in a series of 2D axial CT scans [ 26 ]. These landmarks, positioned at or around the central level of the middle ear, possess unique graphical appearances recognizable by the object detection model. In our recent study, this model demonstrated a 100% success rate in identifying the middle ear region from temporal bone CT scans [ 22 ]. Subsequently, a 3D data matrix (150 × 150 × 32) of the ROI is extracted based on the center coordinates of these 2 structures on each side.

Classification Network

A 3D CNN model is built to interpret the extracted ROI and classify different types of conditions ( Figure 2 B). This model features 4 convolution blocks and 2 dense blocks ( Table 2 ). Each convolution block consists of a 3D convolutional layer to summarize graphical features along all axes of the input image, followed by a max-pooling layer for downsampling these features and another layer for batch normalization. These high-level features are then pooled and passed to the fully connected layers of the dense blocks, where the diagnosis is predicted based on the calculated probability of each class by a softmax function. A dropout layer is applied to prevent overfitting [ 27 ].

a Conv3D: 3D convolutional layer.

b MaxPooling3D: 3D max pooling layer.

c BatchNormalization: batch normalization layer.

d GlobalAveragePooling3D: layer performing global average pooling for 3D data.

Model Training and Testing

Task 1—detection of com.

The first classification model was trained in a binary task distinguishing between normal and pathological ears in all cases (n=3153). The training and testing procedures involved 5-fold cross-validation on the internal (EENT) data set. Specifically, the data set was evenly partitioned into 5 nonoverlapping subsets in a random, stratified fashion. In each iteration, 1 subset was reserved for testing (n=631), while the remaining 4 were used for training (n=2522). Model performance metrics were averaged over 5 iterations of this process. During each training session, a random 20% of training images (n=504) were allocated for validation. Training was set for 1000 epochs with an initial learning rate of 0.0001, and the Adam optimizer was used to dynamically adjust the algorithm’s learning capability and minimize errors [ 28 ]. Early termination was implemented if no further decrease in validation loss was observed for a consecutive 10 epochs. These hyperparameters were determined based on the resultant model performance and training efficiency shown in a preliminary study. The trained model was also evaluated on the external data set (n=211) in each round.

Task 2—Identification of Cholesteatoma

The second classification model was trained to specifically identify cholesteatoma on selected CT scans that displayed signs of inflammation in the middle ears. This task was designed to simulate a common clinical scenario where clinicians need to differentiate cholesteatoma from other types of COM in patients with positive imaging findings. The aim was to provide a preoperative assessment of the risk of cholesteatoma, assisting clinicians in surgical planning [ 3 , 29 ]. For this task, a subset of CT scans with visible soft tissue density or increased opacification in the middle ear or mastoid was selected from both the internal (n=1986) and external sets (n=106). The remaining methods, including extraction of ROI, network architecture, and the training and testing procedures, were consistent with those used in the first task.

Ablation Study

To refine model selection and gain a better understanding of the network’s behavior, an ablation study was performed to compare the proposed classification network with 3 alternative models, each incorporating modifications to certain features. Specifically, the number of convolutional blocks was decreased and increased by 1 in alternative model 1 and model 2, respectively, and a different size of filter was applied in model 3 (Tables S1-S3 in Multimedia Appendix 1 ). To ensure adequate statistical power for detecting differences across models, experiments were conducted on the main data set using the same methodology as outlined in the preceding sections.

Benchmarking Against the 2D Approach

To investigate whether the use of 3D CT scans may enhance diagnostic performance, a benchmark study was designed to compare the proposed system with a baseline model using 2D images. This baseline model, previously established by our team, uses transfer learning on a pretrained Inception-V3 (Google LLC) model [ 22 ]. In this study, the base model of Inception-V3 was retained, and the final classification layer was customized with a binary output. Training and validation were conducted in the same manner as the 3D model, except that only a single CT scan at the central layer of the ROI was used as the input for the 2D model. All image-preprocessing techniques and hyperparameter settings remained consistent with those outlined in the previous study [ 22 ].

Benchmarking Against Human Experts

Another benchmark test was performed against human experts to provide an additional unbiased evaluation of the proposed system. Seven human specialists with a broad range of qualifications were recruited to perform both tasks based on the same image data. The participants included 2 senior otologists, each with 12 years of clinical experience, 1 senior head and neck radiologist with 21 years of experience, 1 attending otologist with 7 years of experience, and 3 otolaryngology residents with 3, 3, and 2 years of experience, respectively. Each expert was provided only with the CT scans and instructed to make a task-specific diagnosis to each ear (task 1: normal or pathological; task 2: cholesteatoma or noncholesteatoma). The test data for clinicians comprised a random selection of 244 ears from the EENT set and all eligible ears from the WU set. To assess intrarater reliability, a random replication of 10% of test cases (n=48) was mixed with these data. All test cases (N=502) had not been previously seen by any experts. They were anonymized, shuffled, and stored on a password-protected computer along with spreadsheets to record each expert’s diagnoses for these cases.

Generation of Heat Maps

Gradient-Weighted Class Activation Mapping was used to visualize model’s rationale for decision-making ( Figure 2 C). In essence, this approach leverages the gradients of the target class flowing into the final convolutional layer to produce a coarse localization heat map, highlighting the critical regions in the image [ 30 ]. In this study, heat maps were generated in a 3D fashion and rescaled to match the original images using TensorFlow 2.11 in Python 3.91 (Python Core Team) [ 31 ].

Clinical Applications

The validated model was integrated into a Python program, enabling the automated assessment of COM from raw CT inputs to the generation of explainable diagnoses in an end-to-end fashion (see the section “Data Availability Statements” and Multimedia Appendix 2 ). To evaluate its viability in assisting otologists in clinical settings, this system was used with a prospective cohort of patients undergoing middle ear surgeries at EENT hospital from November 2023 to January 2024 in a single-arm observational study. Preoperative model predictions, along with routine assessments, were provided to 2 senior otologists, who were given autonomy to determine surgical strategies based on their discretion. Surgeons were surveyed regarding the use of model-generated information in their decision-making processes for these cases. Model predictions were used to analyze the selection of surgical approaches and to measure model performance against pathological findings. Hearing gain was assessed by comparing the air conduction threshold at 2 weeks postoperatively with the baseline.

Statistical Analysis

Descriptive statistics were applied as appropriate. The overall predictability of a model was evaluated by the area under the receiver operating characteristic (AUROC) curve. The optimal cutoff threshold on the curve was determined at the point with minimal distance to the upper left corner on the validation set and subsequently applied to the test set. The numbers of correctly and incorrectly classified cases were displayed in a confusion matrix, and these were used to calculate the performance metrics, including accuracy, recall, specificity, precision, and F 1 -score. These metrics offer comprehensive insights into the model’s performance, covering overall correctness in identifying both positives and negatives (accuracy), sensitivity in detecting positive cases (recall), capability in ruling in patients (specificity), propensity for preventing false alarms (precision), and effectiveness in identifying positive cases while minimizing false positives and false negatives ( F 1 -score). They were derived as shown in Textbox 1 . Results are averaged over 5 iterations of cross-validation or external validation and presented as mean (SD). Intrarater consistency was evaluated using Cohen kappa. Significance was determined through pairwise 2-tailed t test for difference in performance between models and via 1-way analysis of variance between the proposed model and human experts. The alpha level was set at .05. Statistical analyses were conducted using Python 3.91 [ 31 ].

Accuracy = (True positive + True negative)/Total sample size

Recall = True positive/(True positive + False negative)

Specificity = True negative/(True negative + False positive)

Precision = True negative/(True negative + False negative)

F 1 -score=2 × True positive/(2 × True positive + False positive + False negative)

ROI Extraction

The region proposal network successfully extracted the 3D ROI containing the critical anatomies on each side, including the tympanic cavity and sinus tympani ( Figure 3 ). This has been confirmed by manual inspection of the generated images in all cases from both data sets.

Our model exhibited decent performance in identifying pathological changes in the middle ear, achieving a mean accuracy of 87.8%, recall of 85.3%, specificity of 91.3%, and precision of 93.3% on the internal data set ( Table 3 ). It also demonstrated a near-perfect AUROC score of 0.96. These performance metrics remained generally consistent on the external data set, with a comparable AUROC score of 0.93, indicating reasonable generalizability ( Figure 4 ).

a AUROC: area under the receiver operating characteristic curve.

b EENT: Eye, Ear, Nose, and Throat Hospital of Fudan University.

c N/A: not applicable.

d WU: Wuhan Union Hospital.

This model also demonstrated satisfactory predictive capabilities in differentiating between cholesteatoma and noncholesteatomatous cases. On both data sets, the model managed to correctly identify whether a case involved cholesteatoma in approximately 4 out of 5 instances (with accuracies of 78.3% and 81.3%). Generalizability was further supported by the comparable AUROC scores of 0.85 and 0.83 on the internal and the external data set, respectively ( Table 3 ).

This model exhibited a reasonable balance between predictability and efficiency ( Table 4 ). Compared with models 1 and 3, it achieved significantly better performance in both tasks ( P <.01). In addition, despite having approximately 60% fewer parameters, the proposed model demonstrated equivalent performance to model 2 in both tasks ( P =.26 and .91, respectively), indicating its enhanced computational efficiency.

b N/A: not applicable.

Compared with the 2D approach, the 3D network demonstrated significantly superior performance in both tasks across data sets ( P ≤.05). In particular, the proposed model exhibited a substantial performance gain in differentiating between cholesteatoma and noncholesteatomata, with an increase of more than 10% in all outcome metrics on both data sets ( Table 3 ).

This model also matched or even surpassed the diagnostic capabilities of human experts in both tasks ( Figure 4 ). It exhibited marginally superior performance compared with human eyes in the first task ( P =.05) and significantly outperformed them in the visually challenging task 2 ( P <.001). Post hoc pairwise comparisons revealed that the model excelled over the attending otologist in task 1 and 2 resident fellows in task 2, rivaling all senior clinicians ( Table 5 ). Similar results were shown across the breakdown of data sources, with a notable finding that the model outperformed a senior otologist in task 2 on the EENT subset (Table S4 in Multimedia Appendix 1 ). Moreover, the proposed model demonstrated perfect consistency, surpassing all human experts who exhibited higher SDs in all outcome metrics and lower scores of intrarater reliability.

a N/A: not applicable.

b Y: years of experience in clinical practice.

Visual Assessment of Heat Maps

Heat Maps from both models consistently highlighted the tympanic cavity and mastoid that manifested pathological findings characteristic of the target condition ( Figure 5 ). Specifically, the first model generated a hot signal indicative of soft tissue density in an affected middle ear ( Figure 5 A), while the signal remained subdued in a normal ear ( Figure 5 B). Similarly, the second model revealed a distinct hot spot in a cholesteatomatous ear exhibiting the classic patterns of tympanum widening and ossicular destruction [ 17 , 32 , 33 ] ( Figure 5 C). In contrast, a case of CSOM showing intact ossicles surrounded by soft tissue shadows in a normal-sized tympanic cavity did not exhibit a corresponding hot spot ( Figure 5 D). These observations reflect that the AI’s decision-making strategy aligns reasonably well with established human knowledge for both tasks.

Clinical Use

The automatic evaluation system, incorporating the validated 3D model and the heatmap visualization technique, was evaluated for its viability in aiding preoperative assessment in 121 patients with COM (mean age 46.8, SD 16.1 years, 40.5% male). This system achieved an overall accuracy of 81.8% in distinguishing between cholesteatoma and noncholesteatoma cases. Sixty-nine ears were identified as free of cholesteatoma by the model, all of which received minimally invasive tympanoplasty under endoscopy. During the procedure, 9 ears (13.0%) revealed signs of cholesteatoma, and 5 of them required additional bone-grinding technique for complete removal of the mass. Cholesteatoma was initially predicted in 52 ears, with 37 (71.2%) of them undergoing canal-wall-down mastoidectomy. In the remaining 15 ears, the treating surgeons opted for endoscopic tympanoplasty, overriding the conventional technique for the model’s predicted diagnosis. Clinicians reported that the model predictions aligned with their initial judgment or helped with their decision-making in 90.1% (109/121) cases. Postoperative hearing results were obtained in 87.6% (106/121) patients who maintained follow-up. Both groups of ears showed normal recovery, with a mean hearing gain of 8.5 (SD 15.6) and 5.5 (SD 18.1) dB, respectively ( Figure 6 ).

Principal Results

This study demonstrates the robustness and generalizability of an AI model based on 3D CNN for the detection and differential diagnosis of COM using temporal bone CT scans. This model leverages multidimensional diagnostic information from the middle ear, resulting in a significant performance improvement compared with the traditional 2D approach. The framework exhibits comparable or even superior performances to human experts in otologic tasks with clinical significance and visual challenges, especially for classifying between cholesteatoma and noncholesteatomatous cases. In addition, the novel heatmap technique allows inspection of the AI’s logic for decision-making, thereby enhancing the transparency of this model. The resulting AI system serves to automate summarization of critical radiologic findings and enables efficient evaluation of COM with minimum manual input. It provides tangible benefit in assisting otologists during preoperative assessment and results in favorable clinical outcomes that are comparable with historical results [ 34 - 37 ]. These findings further support the clinical viability and advantages of AI technology, which is expected to improve efficiency, reduce errors, and facilitate precision medicine in health care in the new era of big data.

Comparison With Prior Work

A few AI models have recently been developed to classify common middle ear conditions, such as CSOM, otitis media with effusion, and cholesteatoma [ 38 - 41 ]. However, these models were primarily based on traditional otoscopic images, which are potentially limited by a narrow field of view and insufficient diagnostic information. Temporal bone CT scans, which are increasingly used in otologic workup by virtue of its accessibility, rich amount of anatomical information, and adequate sensitivity in revealing pathological changes, have also been explored in a limited number of studies [ 22 , 42 - 45 ]. Although these AI models demonstrated decent AUROC scores (eg, >0.9) in common classification tasks, they were all trained to generate predictions based on 2D single-layer CT scans. A potential drawback is the increased likelihood of missing small or peripheral pathological changes (eg, an attic cholesteatoma) and the resultant false negatives.

Efforts were made in this study to establish a 3D approach to take full advantage of all available anatomical information and achieve a better coverage of the tympanum and the mastoid. Inspection of the extracted ROI suggests that all critical anatomies are visible. Results from the benchmark test indicate that the proposed 3D model outperforms the state-of-the-art 2D approach by a modest performance gain in the detection of COM and by a much larger extent in differentiating between cholesteatoma and noncholesteatoma. This finding has several implications. First, both models are generally adequate in identifying common abnormal patterns from the CT, which are graphically characterized by increased opacification or soft tissue shadows in the middle ear cavity and indicative of pathological conditions in general. This is a relatively simple visual task, during which diagnostic information obtained from a single 2D CT slice is likely sufficient for the purpose and extra findings from other layers provide only minimal contribution to the decision-making. Second, the 3D model has huge advantage over the 2D approach in differentiating cholesteatoma from other types of COM. This task is known to be more visually challenging for humans, often requiring detection of subtle osseous erosions from multiple CT slices, as quite a few pathological changes caused by cholesteatoma are peripheral or noncharacteristic [ 32 , 33 ]. A substantial increase in each outcome measure justifies the advantage of the current 3D model for this task. Moreover, this 3D model has only a simple network structure with a small size (14.5 MB) as opposed to a complex and large-sized 2D network (274 MB), suggesting both higher computational efficiency and performance of the 3D approach. Finally, the AUROC of 0.92-0.94 and accuracy scores of 82.1%-86.1% achieved by the 2D network in this study in detecting COM were equivalent to historical results (0.92%-86%, respectively) in our previous study [ 22 ], further indicating the reliability of these findings and potentially the intrinsic limit of using single-layer CT scan for this task. To the best of our knowledge, this is the first study showing quantitative evidence to support the advantage of a 3D CNN model in 2 common otologic tasks based on temporal bone CT scans. It also advances beyond prior retrospective research by showcasing the practicality and benefits of the model in a clinical environment.

Clinical Implications

Cholesteatoma exhibits distinct histology marked by local invasiveness and a propensity for recurrence. The imperative for successful outcomes necessitates complete removal of the mass, particularly because recurrent cholesteatoma complicates revision surgery [ 46 ]. Suspected cases often require a canal-wall-down mastoidectomy to expose the tympanum, resulting in an open cavity and a permanently altered sound conduction pathway [ 46 ]. Accumulating evidence suggests that noncholesteatoma may spare from mastoidectomy and benefit from minimally invasive procedures such as endoscopic tympanoplasty [ 47 , 48 ]. Therefore, the current AI system holds potential value for otologists in surgical planning. Ears with a low risk of cholesteatoma, as identified by the model, could potentially be treated by less invasive procedures that retain the integrity of canal wall, leading to reduced procedural time and enhanced recovery [ 6 , 7 , 9 , 49 , 50 ]. This clinical merit is supported by the superior benchmark performance in identifying cholesteatoma and the favorable outcomes observed in the prospective study.

While detecting COM in task 1 involves spotting any pathological patterns on CT, which may not fully capture the differences between models in diagnostic capabilities, the increased visual challenges in identifying cholesteatoma substantiate the advantages of the proposed 3D approach for this task. In this study, the 3D model outperformed junior clinicians and demonstrated equivalent or superior performance to senior experts in identifying cholesteatoma based on CT. Notably, the 3D model achieved outcomes that were on par with or better than those based on human interpretation of MRI, which, despite its higher sensitivity, is a more expensive diagnostic method [ 22 , 43 , 45 , 51 - 53 ]. These findings underscore the 3D model’s potential as a reliable and cost-effective alternative, offering sufficient COM evaluation with CT alone, thereby reducing the need for the pricier MRI.

The findings from the prospective study indicate that the model is efficacious in clinical environments, especially in distinguishing cholesteatoma from noncholesteatoma. Feedback from our clinical team highlights that the system serves as a reliable and streamlined source for a second opinion. Before surgery, the treating physician can rapidly identify essential details such as the lesion’s location and properties, using the model’s diagnostic output, and heatmaps. Concordance between the model’s predictions and the physician’s initial assessment bolsters confidence in surgical planning, thereby streamlining the diagnostic and therapeutic process. In contrast, discrepancies between the model’s results and the physician’s judgment prompt a detailed case reassessment or team consultation, aiding in the validation of a suitable treatment plan or preparing for intraoperative modifications. This process provides timely advisory support for complex cases, encouraging meticulous evaluation by the physician, minimizing errors, and keeping the clinician’s cognitive load in check without compromising their autonomy in decision-making.

It should be noted that even for seasoned otologists and radiologists, who are adept at quickly and accurately reading temporal bone CT scans, a second opinion can add an extra layer of confidence to their assessments. For novice clinicians, who may find the diagnostic process more challenging and time-intensive ( Table 5 ), the model may offer substantial improvements in both the accuracy and the speed of diagnosing and managing COM. This is particularly beneficial for physicians in smaller medical facilities or those early in their careers. Looking ahead, the integration of this model into electronic medical systems or cloud-based servers stands to streamline the provision of immediate second opinions or enable physicians from diverse locations to upload imaging data for dependable diagnostic insights. Such technological progress is poised to advance individualized COM treatments in the big data era, boosting efficiency, reducing costs, and enhancing the quality of patient care.

Research Insights

Efforts were undertaken in this study to demystify the criticized nontransparency of DL models, characterized by intricate decision-making strategies within multilayer architectures [ 30 , 54 ]. The nonlinear interactions among these components can yield incomprehensible logic and untraceable predictions vulnerable to bias or errors, posing a significant challenge to the widespread application of AI in health care. To address this issue, heatmaps, and specifically, the Gradient-Weighted Class Activation Mapping technique, have been used as a method to inspect AI’s strategy and enhance human interpretation in a parsimonious manner [ 55 - 57 ]. In this study, the strategy learned by our models to focus on the middle ear and mastoid regions appeared reasonable and aligned with human knowledge in interpreting CT for COM, reinforcing the reliability of this framework. These informative heatmaps can aid clinicians in understanding and validating AI predictions for specific cases, or serve as educational tools for training medical students or junior residents in reading temporal bone CT scans. Ultimately, this approach presents a viable solution for developing explainable AI models for clinical tasks.

Overfitting is a common concern with DL models, especially when data are limited or sourced from a single institute. It can lead to poor performance on new data despite promising results on the original data set. Previous DL models were trained on monocentric CT scans with participant counts ranging from 61 to 562. Lack of external validation and small sample sizes may raise concern about potential overfitting of these models [ 22 , 42 , 43 ]. Several approaches were used in this study to enhance the generalizability of our framework. First, our models underwent cross-validation on a major data set comprising more than 3000 ears, the largest sample size reported to date. Second, these models were evaluated on external data with different patient origins and image properties. Third, several machine learning methods were applied to minimize the risk of models being tuned to the random features, including early termination of training and the use of a dropout function to decrease the interdependency among network nodes [ 27 ]. Consistent performance metrics across data sets in both tasks substantiated the generalizability of this framework. Moreover, the region proposal method proved applicable to CT scans from both sources, demonstrating adaptability despite differences in CT scanner, scan settings, and image quality.

Limitations

This study has several limitations. First, although an external data set was obtained from a hospital in a different city, patients in both data sets shared a common racial background. Further validation on data collected from patients with diverse origins may be necessary to ensure the generalizability of these models. Second, the research was constrained to 2 binary classification tasks relevant to COM. Incorporating additional diagnostic tasks, such as assessing the ossicular chain’s integrity and forecasting auditory outcomes, may enrich the diagnostic toolkit. Third, the models were exclusively trained to analyze CT scans, potentially not leveraging AI’s full potential in COM evaluation. Comprehensive diagnostics often involve synthesizing information from patient history, clinical symptoms, ear examinations, audiological testing, otoscopy, and various imaging techniques. Overreliance on CT scans alone may introduce limitations in performance and may not always lead to conclusive diagnoses ( Figure 7 ). Fourth, the ablation study examined a limited array of model alternatives. Despite achieving notable performance through initial model structure refinement, future endeavors should include ongoing optimization of the model architecture and detailed analysis of network component functions to optimize the trade-off between model efficacy and computational demands. In addition, this study did not place extensive emphasis on exploring common ethical issues, such as patient privacy, data security, and human autonomy, which are critical considerations in the clinical application of AI and warrant ongoing attention. Finally, this study reported initial findings from the clinical application of the AI system in a small, prospective cohort without a control group. Although the main objective was to show that the current model is ready for clinical implementation, a thorough assessment of the model’s clinical benefits will be conducted in an upcoming clinical trial with a more rigorous research design.

Future Research

Future studies will focus on leveraging novel techniques to enhance model performance and evaluate the effectiveness in larger-scale controlled trials. For example, new models will be trained to perform additional tasks, including evaluation of ossicular chain and forecasting postoperative hearing, which may enhance features of the current AI framework. A broader data set will be compiled from hospitals worldwide to assess and refine the generalizability of these models. Moreover, future models will potentially incorporate multiple sources of clinical information with a fusion layer for generating predictions, mimicking human decision-making strategies, and potentially enhancing model robustness. Ongoing efforts will also be made to refine model architectures and to address ethical issues associated with the use of AI in health care. An active learning framework may be established to integrate feedback loops, allowing clinicians to provide input to the model. This approach is expected to support ongoing model enhancement and reinforcement learning based on human feedback. In the next stage, multicenter, prospective human trials will be conducted to assess the practical benefits of implementing these AI models in clinical contexts. The ultimate goal of this research line is to establish a robust AI system that can assist clinicians with reliability, efficiency, and transparency in the evaluation and management of ear diseases.

Conclusions

This study presents a 3D CNN model trained to detect pathological changes and identify cholesteatoma based on temporal bone CT scans. The model’s performance significantly surpasses the baseline 2D approach, reaching a level comparable with or even exceeding that of human experts in both tasks. The model also exhibits decent generalizability and enhanced comprehensibility through the gradient heatmaps. The resulting AI system allows automatic assessment of COM and shows promising viability in real-word clinical settings. These findings imply the potential of AI as a valuable tool for aiding clinicians in the evaluation of COM. Future research will involve enhancing models with additional source of diagnostic information to perform various clinical tasks and evaluating the benefits of AI models in large-scale controlled trials.

Acknowledgments

This study is supported by the National Natural Science Foundation of China (grants 81970889 to Fanglu Chi and 82271166, 81970880, and 81771017 to Dongdong Ren, U22A20249, 52188102, 52027806 to Jiajie Guo); Natural Science Foundation of Shanghai (grant 22ZR1410100 to Dongdong Ren); the “Zhuo-Xue Plan” of Fudan University (Dongdong Ren); Heng-Jie special technical support plan (Dongdong Ren); and the Shanghai Outstanding Young Medical Talent Program (Dongdong Ren). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Data Availability

The complete source code and trained models are available in Multimedia Appendices 1 and 2 as well as in a public repository, which can be accessed in GitHub [ 58 ]. The automatic evaluation system is available for individual use with a detailed instruction manual and a walk-through tutorial. The data sets generated during this study may be obtained from the corresponding authors upon reasonable request.

Authors' Contributions

YL conceptualized and designed the study, reviewed and analyzed the data, performed computer programming, developed and evaluated the AI models, wrote and edited the manuscript, prepared the figures, and supervised the entire project; BC retrieved, validated and analyzed the data, drafted the manuscript, and prepared the figures; YS provided data resources and validated the data. HS, YW, and JL retrieved data and evaluated the models. XN and LY retrieved and validated the data. JG acquired funding support, validated the model, and edited the manuscript. SB performed model validation and deployment. YC, JX, and JY evaluated the models. YH and BL provided data resources; FC provided data resources and funding support; and DR conceptualized the study, provided funding support and data resources, and edited the manuscript. All authors have reviewed, discussed, and approved the manuscript. No generative artificial intelligence tool was used during the preparation of this manuscript. BC, YL, and YS have contributed equally to this study and are credited as co-first authors. YL, BL, and DR are designated as the corresponding authors. Contact details for correspondence are as follows: Yike Li, Department of Otolaryngology-Head and Neck Surgery, Vanderbilt University Medical Center, 1215 21st Avenue South, Rm. 10410, Medical Center East, Nashville, TN 37232, USA. E-mail: [email protected]. ORCID: 0000-0001-8465-130X; Bo Liang, Department of Radiology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Jiefang Avenue #1277, Wuhan, Hubei, 430022, China. E-mail: [email protected]. ORCID: 0000-0002-3494-4187; DongDong Ren, Department of Otorhinolaryngology, Eye, Ear, Nose and Throat Hospital, 83 Fenyang Road, Shanghai, 200031, China. E-mail: [email protected]. ORCID: 0000-0002-2889-9375.

Conflicts of Interest

YL served as an associate editor for the Journal of Medical Internet Research at the time of manuscript submission. YL has abstained from participating in any peer-reviewed or editorial decision-making processes related to this article.

Additional tables.

Source code for model development and validation.

• Schilder AGM, Chonmaitree T, Cripps AW, Rosenfeld RM, Casselbrant ML, Haggard MP, et al. Otitis media. Nat Rev Dis Primers. 2016;2(1):16063. [ CrossRef ] [ Medline ]
• World Health Organization. Chronic Suppurative Otitis Media: Burden of Illness and Management Options. Geneva, Switzerland. World Health Organization; 2004.
• Lustig L, Limb C, Baden R, LaSalvia M. Chronic Otitis Media, Cholesteatoma, and Mastoiditis in Adults. Waltham, MA (citirano 145 2019). UpToDate; 2018.
• Takahashi M, Motegi M, Yamamoto K, Yamamoto Y, Kojima H. Endoscopic tympanoplasty type I using interlay technique. J Otolaryngol Head Neck Surg. 2022;51(1):45. [ CrossRef ] [ Medline ]
• Ohki M, Kikuchi S, Tanaka S. Endoscopic type 1 tympanoplasty in chronic otitis media: comparative study with a postauricular microscopic approach. Otolaryngol Head Neck Surg. 2019;161(2):315-323. [ CrossRef ]
• Hsu Y, Kuo C, Huang T. A retrospective comparative study of endoscopic and microscopic tympanoplasty. J Otolaryngol Head Neck Surg. 2018;47(1):44. [ CrossRef ]
• Yang Q, Wang B, Zhang J, Liu H, Xu M, Zhang W. Comparison of endoscopic and microscopic tympanoplasty in patients with chronic otitis media. Eur Arch Otorhinolaryngol. 2022;279(10):4801-4807. [ CrossRef ]
• Tsetsos N, Vlachtsis K, Stavrakas M, Fyrmpas G. Endoscopic versus microscopic ossiculoplasty in chronic otitis media: a systematic review of the literature. Eur Arch Otorhinolaryngol. 2020;278(4):917-923. [ CrossRef ]
• Tarabichi M, Ayache S, Nogueira JF, Al Qahtani M, Pothier DD. Endoscopic management of chronic otitis media and tympanoplasty. Otolaryngol Clin North Am. 2013;46(2):155-163. [ CrossRef ]
• Watts S, Flood LM, Clifford K. A systematic approach to interpretation of computed tomography scans prior to surgery of middle ear cholesteatoma. J Laryngol Otol. 2000;114(4):248-253. [ CrossRef ]
• Selwyn D, Howard J, Cuddihy P. Pre-operative prediction of cholesteatomas from radiology: retrospective cohort study of 106 cases. J Laryngol Otol. 2019;133(06):477-481. [ CrossRef ]
• Songu M, Altay C, Onal K, Arslanoglu S, Balci MK, Ucar M, et al. Correlation of computed tomography, echo-planar diffusion-weighted magnetic resonance imaging and surgical outcomes in middle ear cholesteatoma. Acta Otolaryngol. 2015;135(8):776-780. [ CrossRef ]
• Mahmutoğlu AS, Celebi I, Sahinoğlu S, Cakmakçi E, Sözen E. Reliability of preoperative multidetector computed tomography scan in patients with chronic otitis media. J Craniofac Surg. 2013;24(4):1472-1476. [ CrossRef ]
• Pandey AK, Bapuraj JR, Gupta AK, Khandelwal N. Is there a role for virtual otoscopy in the preoperative assessment of the ossicular chain in chronic suppurative otitis media? Comparison of HRCT and virtual otoscopy with surgical findings. Eur Radiol. 2009;19(6):1408-1416. [ CrossRef ] [ Medline ]
• Chee NW, Tan TY. The value of pre-operative high resolution CT scans in cholesteatoma surgery. Singapore Med J. 2001;42(4):155-159.
• Zaman SU, Rangankar V, Muralinath K, Shah V, Pawar R. Temporal bone cholesteatoma: typical findings and evaluation of diagnostic utility on high resolution computed tomography. Cureus. 2022;14(3):e22730. [ CrossRef ] [ Medline ]
• Gaurano JL, Joharjy IA. Middle ear cholesteatoma: characteristic CT findings in 64 patients. Ann Saudi Med. 2004;24(6):442-447. [ CrossRef ] [ Medline ]
• Ardila D, Kiraly AP, Bharadwaj S, Choi B, Reicher JJ, Peng L, et al. End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. Nat Med. 2019;25(6):954-961. [ CrossRef ] [ Medline ]
• Mikhael PG, Wohlwend J, Yala A, Karstens L, Xiang J, Takigami AK, et al. Sybil: a validated deep learning model to predict future lung cancer risk from a single low-dose chest computed tomography. J Clin Oncol. 2023;41(12):2191-2200. [ CrossRef ]
• Yamashita R, Long J, Longacre T, Peng L, Berry G, Martin B, et al. Deep learning model for the prediction of microsatellite instability in colorectal cancer: a diagnostic study. Lancet Oncol. 2021;22(1):132-141. [ CrossRef ] [ Medline ]
• Li Y, Guo J, Yang P. Developing an image-based deep learning framework for automatic scoring of the pentagon drawing test. J Alzheimers Dis. 2022;85(1):129-139. [ CrossRef ]
• Wang Y, Li Y, Cheng Y, He Z, Yang J, Xu J, et al. Deep learning in automated region proposal and diagnosis of chronic otitis media based on computed tomography. Ear Hear. 2020;41(3):669-677. [ CrossRef ]
• Sundgaard JV, Harte J, Bray P, Laugesen S, Kamide Y, Tanaka C, et al. Deep metric learning for otitis media classification. Med Image Anal. 2021;71:102034. [ CrossRef ] [ Medline ]
• Watson DS, Krutzinna J, Bruce IN, Griffiths CE, McInnes IB, Barnes MR, et al. Clinical applications of machine learning algorithms: beyond the black box. BMJ. 2019;364:l886. [ CrossRef ]
• Castelvecchi D. Can we open the black box of AI? Nature. 2016;538(7623):20-23. [ CrossRef ] [ Medline ]
• Redmon J, Divvala S, Girshick R, Farhadi A. You only look once: unified, real-time object detection. IEEE; 2016. Presented at: Proceedings of the IEEE conference on computer vision and pattern recognition; 2016:779-788; Las Vegas, NV, USA. [ CrossRef ]
• Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R. Dropout: a simple way to prevent neural networks from overfitting. J machine Learn Res. 2014;15(1):1929-1958.
• Kingma DP, Ba J. Adam: a method for stochastic optimization. arXiv. 13:54-29. Preprint published online Dec 2014. [ FREE Full text ]
• Tseng C, Lai M, Wu C, Yuan S, Ding Y. Comparison of the efficacy of endoscopic tympanoplasty and microscopic tympanoplasty: a systematic review and meta‐analysis. Laryngoscope. 2016;127(8):1890-1896. [ CrossRef ] [ Medline ]
• Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-cam: visual explanations from deep networks via gradient-based localization. 2017. Presented at: Proceedings of the IEEE International Conference on Computer Vision; 2017:618-626; Venice, Italy.
• Python Core Team. Python: a dynamic, open source programming language. Python Software Foundation. URL: https://www.python.org/ [accessed 2024-06-01]
• Baráth K, Huber AM, Stämpfli P, Varga Z, Kollias S. Neuroradiology of cholesteatomas. AJNR Am J Neuroradiol. 2011;32(2):221-229. [ CrossRef ]
• Gulati M, Gupta S, Prakash A, Garg A, Dixit R. HRCT imaging of acquired cholesteatoma: a pictorial review. Insights Imaging. 2019;10(1):92. [ CrossRef ] [ Medline ]
• Daneshi A, Daneshvar A, Asghari A, Farhadi M, Mohebbi S, Mohseni M. Endoscopic versus microscopic cartilage myringoplasty in chronic otitis media. Iran J Otorhinolaryngol. 2020;32(112):263-269. [ CrossRef ]
• Prasad SC, Melia CL, Medina M, Vincenti V, Bacciu A, Bacciu S, et al. Long-term surgical and functional outcomes of the intact canal wall technique for middle ear cholesteatoma in the paediatric population. Acta Otorhinolaryngol Ital. 2014;34(5):354-361.
• Wood CB, O’Connell BP, Lowery AC, Bennett ML, Wanna GB. Hearing outcomes following type 3 tympanoplasty with stapes columella grafting in canal wall down mastoidectomy. Ann Otol Rhinol Laryngol. 2019;128(8):736-741. [ CrossRef ]
• Chamoli P, Singh CV, Radia S, Shah AK. Functional and anatomical outcome of inside out technique for cholesteatoma surgery. Am J Otolaryngol. 2018;39(4):423-430. [ CrossRef ] [ Medline ]
• Wu Z, Lin Z, Li L, Pan H, Chen G, Fu Y, et al. Deep learning for classification of pediatric otitis media. Laryngoscope. 2020;131(7):E2344-E2351. [ CrossRef ]
• Pichichero ME. Can machine learning and AI replace otoscopy for diagnosis of otitis media? Pediatrics. 2021;147(4):e2020049584. [ CrossRef ]
• Tseng CC, Lim V, Jyung RW. Use of artificial intelligence for the diagnosis of cholesteatoma. Laryngoscope Invest Otolaryngol. 2023;8(1):201-211. [ CrossRef ]
• Livingstone D, Chau J. Otoscopic diagnosis using computer vision: an automated machine learning approach. Laryngoscope. 2020;130(6):1408-1413. [ CrossRef ] [ Medline ]
• Duan B, Guo Z, Pan L, Xu Z, Chen W. Temporal bone CT-based deep learning models for differential diagnosis of primary ciliary dyskinesia related otitis media and simple otitis media with effusion. Am J Transl Res. 2020;14(7):4728-4735.
• Eroğlu O, Eroğlu Y, Yıldırım M, Karlıdag T, Çınar A, Akyiğit A, et al. Is it useful to use computerized tomography image-based artificial intelligence modelling in the differential diagnosis of chronic otitis media with and without cholesteatoma? Am J Otolaryngol. 2022;43(3):103395. [ CrossRef ]
• Khosravi M, Jabbari Moghaddam Y, Esmaeili M, Keshtkar A, Jalili J, Tayefi Nasrabadi H. Classification of mastoid air cells by CT scan images using deep learning method. J Big Data. 2022;9(1):1-14. [ CrossRef ]
• Wang Z, Song J, Su R, Hou M, Qi M, Zhang J, et al. Structure-aware deep learning for chronic middle ear disease. Expert Syst Appl. 2022;194:116519. [ CrossRef ]
• Tomlin J, Chang D, McCutcheon B, Harris J. Surgical technique and recurrence in cholesteatoma: a meta-analysis. Audiol Neurotol. 2013;18(3):135-142. [ CrossRef ]
• Lee S, Lee DY, Seo Y, Kim YH. Can endoscopic tympanoplasty be a good alternative to microscopic tympanoplasty? a systematic review and meta-analysis. Clin Exp Otorhinolaryngol. 2019;12(2):145-155. [ CrossRef ] [ Medline ]
• Trinidade A, Page JC, Dornhoffer JL. Therapeutic mastoidectomy in the management of noncholesteatomatous chronic otitis media. Otolaryngol Head Neck Surg. 2016;155(6):914-922. [ CrossRef ] [ Medline ]
• Wu L, Liu Q, Gao B, Huang S, Yang N. Comparison of endoscopic and microscopic management of attic cholesteatoma: a randomized controlled trial. Am J Otolaryngol. 2022;43(3):103378. [ CrossRef ] [ Medline ]
• Toulouie S, Block‐Wheeler NR, Rivero A. Postoperative pain after endoscopic vs microscopic otologic surgery: a systematic review and meta-analysis. Otolaryngol Head Neck Surg. 2022;167(1):25-34. [ CrossRef ] [ Medline ]
• Profant M, Sláviková K, Kabátová Z, Slezák P, Waczulíková I. Predictive validity of MRI in detecting and following cholesteatoma. Eur Arch Otorhinolaryngol. 2012;269(3):757-765. [ CrossRef ]
• Lin M, Sha Y, Sheng Y, Chen W. Accuracy of 2D blade turbo gradient- and spin-echo diffusion weighted imaging for the diagnosis of primary middle ear cholesteatoma. Otol Neurotol. 2022;43(6):e651-e657. [ CrossRef ] [ Medline ]
• Sharifian H, Taheri E, Borghei P, Shakiba M, Jalali AH, Roshanfekr M, et al. Diagnostic accuracy of non‐echo‐planar diffusion‐weighted MRI versus other MRI sequences in cholesteatoma. J Med Imaging Radiat Oncol. 2012;56(4):398-408. [ CrossRef ] [ Medline ]
• Montavon G, Lapuschkin S, Binder A, Samek W, Müller K. Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recognit. 2017;65:211-222. [ CrossRef ]
• Panwar H, Gupta PK, Siddiqui MK, Morales-Menendez R, Bhardwaj P, Singh V. A deep learning and grad-CAM based color visualization approach for fast detection of Covid-19 cases using chest X-ray and CT-scan images. Chaos Solitons Fractals. 2020;140:110190. [ CrossRef ] [ Medline ]
• Cheng C, Ho T, Lee T, Chang C, Chou C, Chen C, et al. Application of a deep learning algorithm for detection and visualization of hip fractures on plain pelvic radiographs. Eur Radiol. 2019;29(10):5469-5477. [ CrossRef ] [ Medline ]
• He T, Guo J, Chen N, Xu X, Wang Z, Fu K, et al. MediMLP: using grad-CAM to extract crucial variables for lung cancer postoperative complication prediction. IEEE J Biomed Health Inform. 2020;24(6):1762-1771. [ CrossRef ] [ Medline ]
• huntlylee / 3D-Otitis-Media. GitHub. URL: https://github.com/huntlylee/3D-Otitis-Media [accessed 2024-08-01]

Abbreviations

Edited by S Ma; submitted 09.08.23; peer-reviewed by Q Chen, SA Javed, CN Hang; comments to author 02.11.23; revised version received 30.11.23; accepted 29.05.24; published 08.08.24.

©Binjun Chen, Yike Li, Yu Sun, Haojie Sun, Yanmei Wang, Jihan Lyu, Jiajie Guo, Shunxing Bao, Yushu Cheng, Xun Niu, Lian Yang, Jianghong Xu, Juanmei Yang, Yibo Huang, Fanglu Chi, Bo Liang, Dongdong Ren. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 08.08.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Research on path planning technology of a line scanning measurement robot based on the cad model.

1. Introduction

2. materials and methods, 2.1. line laser scanner scanning measurement, 2.1.1. measuring principle of the line laser scanner, 2.1.2. measurement constraints of the line laser scanner, 2.1.3. line laser scanner scan path composition, 2.2. free-form scan path planning method, 2.2.1. adaptive sampling method, adaptive sampling methods of free curves, adaptive sampling method for laser line scan of free-form surfaces, 2.2.2. scanning viewpoint planning based on viewable cones, 2.2.3. quaternion-based scanning attitude calculation, 2.2.4. scan path generation based on bi-directional scanning, 3. experiments and discussions, 3.1. simulation of scanning measurement processes, 3.2. robotic 3d scanning and point cloud reconstruction experiments, 4. conclusions, author contributions, data availability statement, conflicts of interest.

• Wang, Z.; Yin, M.; Ou, D.; Xie, L.; Liu, H.; Yin, G. Optical Measurement Method for Blade Profiles Based on Blade Self-Features. IEEE Trans. Ind. Electron. 2022 , 69 , 2067–2076. [ Google Scholar ] [ CrossRef ]
• Yang, S.; Jiang, X.; Xu, S.; Wang, X. Information theory based probabilistic approach to blade damage detection of turbomachine using sensor data. IEEE Trans. Ind. Electron. 2020 , 67 , 10887–10896. [ Google Scholar ] [ CrossRef ]
• Li, W.L.; Wang, G.; Zhang, G.; Li, Q.D.; Yin, Z.P. Interferencefree inspection path generation for impeller blades using an on-machine probe. IEEE/ASME Trans. Mechatron. 2017 , 22 , 1218–1226. [ Google Scholar ] [ CrossRef ]
• Bi, Q.; Huang, N.; Zhang, S.; Shuai, C.; Wang, Y. Adaptive machining for curved contour on deformed large skin based on on-machine measurement and isometric mapping. Int. J. Mach. Tools Manuf. 2019 , 136 , 34–44. [ Google Scholar ] [ CrossRef ]
• Glorieux, E.; Franciosa, P.; Ceglarek, D. Coverage path planning with targetted viewpoint sampling for robotic free-form surface inspection. Robot. Comput.-Integr. Manuf. 2020 , 61 , 101843. [ Google Scholar ] [ CrossRef ]
• Liu, Y.; Zhao, W.; Sun, R.; Yue, X. Optimal path planning for automated dimensional inspection of free-form surfaces. J. Manuf. Syst. 2020 , 56 , 84–92. [ Google Scholar ] [ CrossRef ]
• Gopinath, V.; Johansen, K.; Derelöv, M.; Gustafsson, Å.; Axelsson, S. Safe collaborative assembly on a continuously moving line with large industrial robots. Robot. Comput.-Integr. Manuf. 2021 , 67 , 102048. [ Google Scholar ] [ CrossRef ]
• Wang, J.; Gong, Z.; Tao, B.; Yin, Z. A 3-D reconstruction method for large freeform surfaces based on mobile robotic measurement and global optimization. IEEE Trans. Instrum. Meas. 2022 , 71 , 5006809. [ Google Scholar ] [ CrossRef ]
• Chen, C.; Jia, H.; Lu, Y.; Zhang, X.; Chen, H.; Yu, L. An Adaptive Hybrid Sampling Method for Free-Form Surfaces Based on Geodesic Distance. Sensors 2023 , 23 , 3224. [ Google Scholar ] [ CrossRef ]
• Babu, A.; Manoj, B.; Pasquale Franciosa, C.; Dariusz Ceglarek, D. Spatio-Temporal Adaptive Sampling for effective coverage measurement planning during quality inspection of free form surfaces using robotic 3D optical scanner. J. Manuf. Syst. 2019 , 53 , 93–108. [ Google Scholar ] [ CrossRef ]
• Pagani, L.A.; Scott, P.J.B. Curvature based sampling of curves and surfaces. Comput. Aided Geom. Des. 2018 , 59 , 32–48. [ Google Scholar ] [ CrossRef ]
• Zahmati, J.A.; Amirabadi, H.B.; Mehrad, V.C. A hybrid measurement sampling method for accurate inspection of geometric errors on freeform surfaces. Measurement 2018 , 122 , 155–167. [ Google Scholar ] [ CrossRef ]
• Yi, B.; Qiao, F.; Huang, N.; Wang, X.; Wu, S.; Biermann, D. Adaptive sampling point planning for free-form surface inspection under multi-geometric constraints. Precis. Eng. 2021 , 72 , 95–101. [ Google Scholar ] [ CrossRef ]
• Gu, K.; Chen, S.; You, X.; Li, Y.; Cui, J.; Wang, Y.; Liu, J. Sparse sampling and reconstruction for freeform surface based on low-rank matrix completion. Surf. Topogr. Metrol. Prop. 2021 , 9 , 045053. [ Google Scholar ] [ CrossRef ]
• Zhang, K.; Wang, W.; Zhao, B.; Chen, Y. A three-dimensional surface measurement system implemented with Gaussian process based adaptive sampling. Precis. Eng. 2021 , 72 , 595–603. [ Google Scholar ] [ CrossRef ]
• Liu, Y.; Zhao, W.; Liu, H.; Wang, Y.; Yue, X. Coverage Path Planning for Robotic Quality Inspection With Control on Measurement Uncertainty. IEEE/ASME Trans. Mechatron. 2022 , 27 , 3482–3493. [ Google Scholar ] [ CrossRef ]
• Feng, G.; Ziyue, P.; Xutao, Z.; Yan, L.; Jihao, D. An adaptive sampling method for accurate measurement of aeroengine blades. Measurement 2021 , 173 , 108531. [ Google Scholar ] [ CrossRef ]
• Zhang, B.; Feng, P.; Sun, Z.; Cheng, X.; Zeng, L.; Fan, C. Efficient sampling method based on co-kriging for free-form surface measurement. Precis. Eng. 2023 , 84 , 136–147. [ Google Scholar ] [ CrossRef ]
• Ding, L.J.; Dai, S.G.; Mu, P.A. CAD-based path planning for 3D laser scanning of complex surface. Procedia Comput. Sci. 2016 , 92 , 526–535. [ Google Scholar ] [ CrossRef ]
• Koutecký, T.; Paloušek, D.; Brandejs, J. Sensor planning system for fringe projection scanning of sheet metal parts. Measurement 2016 , 94 , 60–70. [ Google Scholar ] [ CrossRef ]
• Morozov, M.; Pierce, S.G.; MacLeod, C.N.; Mineo, C.; Summan, R. Off-line scan path planning for robotic NDT. Measurement 2018 , 122 , 284–290. [ Google Scholar ] [ CrossRef ]
• Li, L.; Xu, D.; Niu, L.; Lan, Y.; Xiong, X. A path planning method for a surface inspection system based on two-dimensional laser profile scanner. Int. J. Adv. Robot. Syst. 2019 , 16 , 1729881419862463. [ Google Scholar ] [ CrossRef ]
• Ziegler, J.; Gattringer, H.; Kaserer, D.; Müller, A. Automated, depth sensor based object detection and path planning for robot-aided 3d scanning. In Advances in Service and Industrial Robotics: Proceedings of the 26th International Conference on Robotics in Alpe-Adria-Danube Region, Poitiers, France, 21–23 June 2017 ; Springer: Cham, Switzerland, 2017. [ Google Scholar ]
• Phan, N.D.M.; Quinsat, Y.; Lavernhe, S.; Lartigue, C. Path planning of a laser-scanner with the control of overlap for 3d part inspection. Procedia Cirp 2018 , 67 , 392–397. [ Google Scholar ] [ CrossRef ]
• Wang, J.; Tao, B.; Gong, Z.; Yu, S.; Yin, Z. A mobile robotic measurement system for large-scale complex components based on optical scanning and visual tracking. Robot. Comput.-Integr. Manuf. 2021 , 67 , 102010. [ Google Scholar ] [ CrossRef ]
• Zong, Y.; Liang, J.; Pai, W.; Ye, M.; Ren, M.; Zhao, J.; Tang, Z.; Zhang, J. A high-efficiency and high-precision automatic 3D scanning system for industrial parts based on a scanning path planning algorithm. Opt. Lasers Eng. 2022 , 158 , 107176. [ Google Scholar ] [ CrossRef ]
• Li, J.; Wang, M.; Qu, L.; Lv, G. Research on Three-Dimensional Scanning Path Planning of Casing Parts Based on Industrial Robot. Appl. Sci. 2023 , 13 , 6162. [ Google Scholar ] [ CrossRef ]
• Li, Q.; Huang, X.; Li, S. A laser scanning posture optimization method to reduce the measurement uncertainty of large complex surface parts. Meas. Sci. Technol. 2019 , 30 , 105203. [ Google Scholar ] [ CrossRef ]
• Kwon, H.; Na, M.; Song, J.B. Rescan strategy for time efficient view and path planning in automated inspection system. Int. J. Precis. Eng. Manuf. 2019 , 20 , 1747–1756. [ Google Scholar ] [ CrossRef ]
• Lee, I.D.; Seo, J.H.; Kim, Y.M.; Choi, J.; Han, S.; Yoo, B. Automatic pose generation for robotic 3D scanning of mechanical parts. IEEE Trans. Robot. 2020 , 36 , 1219–1238. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Jia, H.; Chen, H.; Chen, C.; Huang, Y.; Lu, Y.; Gao, R.; Yu, L. Research on Path Planning Technology of a Line Scanning Measurement Robot Based on the CAD Model. Actuators 2024 , 13 , 310. https://doi.org/10.3390/act13080310

Jia H, Chen H, Chen C, Huang Y, Lu Y, Gao R, Yu L. Research on Path Planning Technology of a Line Scanning Measurement Robot Based on the CAD Model. Actuators . 2024; 13(8):310. https://doi.org/10.3390/act13080310

Jia, Huakun, Haohan Chen, Chen Chen, Yichen Huang, Yang Lu, Rongke Gao, and Liandong Yu. 2024. "Research on Path Planning Technology of a Line Scanning Measurement Robot Based on the CAD Model" Actuators 13, no. 8: 310. https://doi.org/10.3390/act13080310

Article Metrics

Further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

• Open access
• Published: 09 August 2024

Exploring Agrobacterium -mediated genetic transformation methods and its applications in Lilium

• Xinyue Fan 1 &
• Hongmei Sun 1 , 2  

Plant Methods volume  20 , Article number:  120 ( 2024 ) Cite this article

19 Accesses

Metrics details

As a typical bulb flower, lily is widely cultivated worldwide because of its high ornamental, medicinal and edible value. Although breeding efforts evolved over the last 10000 years, there are still many problems in the face of increasing consumer demand. The approach of biotechnological methods would help to solve this problem and incorporate traits impossible by conventional breeding. Target traits are dormancy, development, color, floral fragrance and resistances against various biotic and abiotic stresses, so as to improve the quality of bulbs and cut flowers in planting, cultivation, postharvest, plant protection and marketing. Genetic transformation technology is an important method for varietal improvement and has become the foundation and core of plant functional genomics research, greatly assisting various plant improvement programs. However, achieving stable and efficient genetic transformation of lily has been difficult worldwide. Many gene function verification studies depend on the use of model plants, which greatly limits the pace of directed breeding and germplasm improvement in lily. Although significant progress has been made in the development and optimization of genetic transformation systems, shortcomings remain. Agrobacterium -mediated genetic transformation has been widely used in lily. However, severe genotypic dependence is the main bottleneck limiting the genetic transformation of lily. This review will summarizes the research progress in the genetic transformation of lily over the past 30 years to generate the material including a section how genome engineering using stable genetic transformation system, and give an overview about recent and future applications of lily transformation. The information provided in this paper includes ideas for optimizing and improving the efficiency of existing genetic transformation methods and for innovation, provides technical support for mining and identifying regulatory genes for key traits, and lays a foundation for genetic improvement and innovative germplasm development in lily.

Flowers are not only a globally important agricultural industry with great economic benefits but also necessary agents for mental health in people's daily lives. Many countries have given intensive attention to the development of the flower industry. At present, the world's flower cultivation area is 22.3 hm 2 , and the international trade volume of flowers is expanding (AIPH, https://www.floraldaily.com ). Lily is a typical perennial herbaceous bulb plant with more than 100 wild species and more than 9,000 varieties worldwide; additionally, this plant has high ornamental value, and the share of lily as a cut flower in the global flower market is increasing annually [ 1 ]. Some lily varieties are edible and have high medicinal value, and their extracts are rich in antioxidant and anti-inflammatory components, resulting in widespread use in medicine, functional food and cosmetics [ 2 ]. It is therefore unsurprising that lilies are the focus of much bulbous flower research. Improving the quality of seed balls and cut flowers has always been a key goal in the global lily industry [ 3 , 4 ]. The target traits are dormancy, development, colour, floral fragrance and resistance to various biological and abiotic stresses to improve the quality of bulbs and cut flowers in planting, cultivation, postharvest, plant protection and marketing. Crossbreeding can quickly fuse good traits and easily produce heterosis. However, lily has a complex genetic background, high heterozygosity, and it is extremely difficult to carry out genome processing, because it is one of the plants with the largest genome, with nearly 30 Gb of genetic information. The long cycle of cross-breeding requires a lot of manpower and material resources. In many cases, sexual incompatibility is also an obstacle in the crossbreeding of lily. Likewise, physical and chemical mutagenesis methods are also highly uncertain. With the rapid development of molecular biology technology, genetic engineering has received increasing amounts of attention. Using molecular methods to improve germplasms can not only lead to the creation of new traits but also increase the efficiency and accuracy of breeding [ 5 , 6 , 7 ] (Fig.  1 ).

Strategies for developing new varieties of lily using different plant breeding tools. A Physical mutagenesis (in which mutants are created by exposing seeds or bulblets to radiation). B Chemical mutagenesis (treatment of different explants with chemical agents to obtain mutants, such as EMS mutagenesis). C Traditional crossbreeding has led to the cultivation of new lily varieties. D Transgenic breeding

Genetic transformation is an important part of genetic engineering technology, and the main goals of flower genetic transformation are as follows: (1) genetic transformation for basic research on a single gene, gene family or gene regulatory network and (2) the application of basic research results as the theoretical basis for improving flower traits and creating new varieties. Plant genetic transformation includes target gene selection, delivery, integration into plant cells, and expression and, ultimately, the production of a complete plant after numerous processes [ 8 ]. Although genetically modified plants were obtained in 1983, the genetic transformation of bulb flowers such as lily has long been considered difficult or impossible [ 9 ]. With progress in the field of lily research, an increasing number of genes, including genes related to major factors involved in regulating various life activities, responding to various biological and abiotic stresses, and responding to various environmental signals, have been identified. However, many gene function studies still depend on the heterologous transformation of model plants such as Arabidopsis and Nicotiana benthamiana (Fig. S1) . In 1992, Cohen conducted the first genetic transformation experiment in lily through Agrobacterium -mediated transformation and detected foreign genes in the calli [ 10 ]. However, the low efficiency, difficulty of regeneration, and difficulty of integration into the lily genome remain obstacles to overcome. With increasing basic research on lily plants, instantaneous transformation based on virus induction has gradually become the first choice of many researchers because this method is simple and fast, and the research cycle is only a few hours or days. However, because these methods cannot enable integration into the genome and are sometimes limited to a single tissue, it is difficult to provide sufficient evidence for gene function, and the research results are uninformative and unfavourable for further application in breeding work. Agrobacterium -mediated transformation, particle bombardment, PEG and electric shock are common methods of plant genetic transformation at present (Fig.  2 ). Compared with other plant transgenic methods, Agrobacterium -mediated plant genetic transformation remains the most common and widespread lily transgenic strategy because of its advantages of high transformation efficiency, few transgenic copies, and stable transfer of integrated genes into offspring after continuous optimization and updating [ 11 ].

Common methods for the genetic transformation of lily. A Agrobacterium -mediated stable genetic transformation B Virus-induced transient gene silencing (VIGS). C Particle bombardment. D Pollen magnetic effect method

In the past 30 years, many researchers have attempted to improve and create technology by adjusting or changing various parameters and operating methods and have accumulated considerable valuable experience (Fig.  3 ). Agrobacterium is a gram-negative bacterial genus that is widely distributed in soil. At the beginning of the twentieth century, the principle that natural pathogens can infect plants through wounds was gradually elucidated. The main mechanism is the delivery of tumorigenic DNA molecules (transfer DNA or T-DNA) into plant cells through wounds in infected plants; these molecules are eventually integrated into the host genome and stably transmitted to the next generation of the plant through meiosis [ 12 , 13 ]. The ability of Agrobacterium to integrate its own DNA into the host genome is primarily determined by the Ti plasmid [ 14 ], which can be modified by the insertion of target genes into the T-DNA region. With the help of the transferability of this region, genes can be introduced into plants by Agrobacterium infection and incorporated into plant genome, after which transgenic plants can be generated by cell and tissue culture technology [ 13 ]. Like most plant genetic transformation methods mediated by Agrobacterium, the genetic transformation procedures for lily mainly include vector construction, selection and culture of explants, preculture, Agrobacterium infection, coculture, resistance selection and transgenic plant regeneration (Fig.  2 ). At present, in addition to calli induced by roots, petals and leaves, scales and embryonic calli are common explants used for genetic transformation in lily [ 15 , 16 , 17 ]. With the continuous updating and optimization of genetic transformation technology for lily, functional verification of several genes by heterologous transformation of model plants has gradually increased, and many genes that regulate desirable traits in lily have been identified [ 3 , 17 , 18 , 19 , 20 ]. Nevertheless, stable and efficient transformation of target genes has been achieved in only a few lily varieties, and the success rate of genetic transformation of some lily varieties is still low.

Timeline of several major discoveries, applications and breakthroughs in the history of lily genetic transformation

In this paper, the development of genetic transformation technology for lily plants over the past 30 years is reviewed, the factors and key technical points restricting the efficiency of genetic transformation in lily are described, the problems and limitations associated with the genetic transformation of lily are summarized, and the prospects for application and improvement are discussed. The purpose of this paper is to provide a technical reference for establishing a stable and efficient genetic transformation system for lily and to lay a foundation for directional breeding and genetic improvement of key characteristics.

Factors affecting the genetic transformation of lily

Many factors affect lily regeneration and transformation. Genotyping is one of the key problems affecting the success of transformation [ 16 ]. Although genetic transformation systems have been established for different lily varieties, major differences exist between different genotypes [ 21 ]. Under the same conditions during the genetic transformation process, the genotype determines the difficulty of using Agrobacterium to successfully infect lily. At present, stable and efficient genetic transformation has been successfully achieved for few lily varieties (Table  1 ). Since the use of Agrobacterium -mediated genetic transformation of lily has been reported, several studies have aimed to optimize the transformation system or establish methods suitable for different lily species, including Lilium formolongi [ 22 , 23 ], Lilium longiflorum [ 24 , 25 , 26 ], Lilium pumilum DC.Fisch. [ 26 ] and the Oriental hybrid Lily [ 15 , 16 , 27 , 28 ]. Due to the strong genotypic dependence and difficult regeneration of explant materials after transformation, most related research results are restricted to certain genotypes [ 16 , 25 , 27 , 28 ]. Yan et al. [ 26 ] established a stable and efficient transformation system through somatic embryogenesis and adventitious bud regeneration in Lilium pumilum DC. Fisch. and Lilium longiflorum . After method optimization, the transformation efficiency reached 29.17% and 4%, respectively. Although the transformation efficiency in 'White Heaven' is still low, it is relatively stable and can be regenerated within 1 month. Song et al. [ 17 ] improved the original transformation system by adjusting the pH and CaCl 2 concentration of the medium; the number of resistant plants increased by 2.7–6.4 times, the number of positive lines increased by 3–6 times transformation, and the genetic transformation efficiency increased by 5.7–13.0%. In the latest study, the genetic transformation system of Oriental hybrid lily was further optimized, and the efficiency was increased to 60% by screening for the lethal concentration of antibiotics, the concentration of the bacterial solution and the duration of infection.

Good explant material is the basis of plant genetic transformation. The success of transformation depends on the selection and totipotency of explants [ 32 ]. Researchers have tested different explants for genetic transformation of target genes based on the Agrobacterium system (Table  1 ). Calli generated from floral organs, scales, leaves, or seeds have been used for genetic transformation in most lily hybrids [ 27 , 33 , 34 ]. Related studies have shown that filamentous calli have a faster growth rate and may be more susceptible to Agrobacterium infection [ 35 ]. In the Oriental hybrid lily ( Lilium cv. Acapulco), an Agrobacterium -mediated lily transformation system was successfully established by using filament-induced filiform calli as explants. Although transient expression of the GUS reporter gene could be detected by root–, leaf–, stalk–, ovary– and anther-induced callus infection, no positive transgenic tissues or plants were obtained [ 27 ]. In recent years, several other explants have been developed and offer additional possibilities for improving transformation efficiency. Liu et al. [ 24 ] discussed the effect of the direct regeneration pathway and the callus regeneration pathway on the transformation efficiency in Agrobacterium -based genetic transformation experiments using stem segments induced by lily scales as explants. Notably, when stem segments were used as explants, adventitious buds obtained via the direct regeneration pathway after coculture significantly increased the regeneration rate of resistant plants and decreased the gene escape rate [ 24 ]. Another study showed that the direct use of scales as transformation explants did not significantly improve the transformation rate but did greatly shorten the genetic transformation cycle [ 26 ]. Different explant materials have their own advantages. Cohen and Meredith [ 10 ] used a particle bombardment approach to carry out lily genetic transformation and reported that the ability of embryonic calli to accept foreign genes was 50–70 times greater than that of ordinary calli. Embryogenic calli are composed of many embryogenic cells, and each cell has the potential to develop into adult somatic embryos. Therefore, using embryogenic calli as explant materials can result in a more stable transformation population with a lower chimaerism rate, which is very important for the research and development of plant genetic transformation [ 36 , 37 , 38 , 39 ]. Mercuri et al. [ 25 ] induced embryonic calli using the styles and peduncles of Lilium longiflorum ‘Snow Queen’, and they were found to be highly competent for transformation. Recently, two studies reported an efficient protocol with high transformation efficiency for Lilium pumilum DC.Fisch. using embryonic calli. Despite their long transformation cycle, embryogenic calli are the most common explant material for the genetic transformation of lily due to their high cell proliferation rate and genetic stability [ 17 , 26 ].

Strains of agrobacterium

To date, many successful cases of stable genetic transformation of plants mediated by Agrobacterium have been reported [ 40 , 41 , 42 , 43 , 44 ]. The strain of Agrobacterium can also considerably influence the transformation frequency. With the continuous updating and optimization of plant genetic transformation technology, several researchers have attempted to improve the efficiency of plant transformation by changing various parameters, including Agrobacterium strains [ 45 ] (Table  1 ). Different plants have different preferences for the routinely used Agrobacterium strains EHA105, EHA101, LBA4404, GV3101, AGL1 and C58 [ 8 , 27 , 29 , 46 ]. In alfalfa [ 47 ], tomato [ 48 ], grasspea [ 49 ], and pigeon pea [ 50 ], LB4404 and LBA4404 were found to be more virulent and highly effective, offering higher transformation efficiency. However, the LBA4404 strain has been less frequently reported in lilies. Mercuri et al. [ 25 ] reported that LBA4404 effectively promoted the infection of embryogenic calli from Lilium longiflorum 'Snow Queen'. According to another study of genetic transformation in lily, the use of the EHA105 strain to infuse embryonic calli seemed to be more beneficial for improving transformation efficiency [ 26 ]. In recent years, the strains EHA101 and EHA105 have been more widely used in lily transformation and are considered to result in greater transformation frequency [ 16 , 18 ].

Selection of markers and reporter genes

The selection of marker genes also determines the efficiency of plant genetic transformation. They are usually delivered along with the target gene, conferring resistance to toxic compounds in plants and facilitating the growth of transformed cells in the presence of such unfavourable conditions. Normally, the marker gene and the target gene are connected to the same plasmid and delivered to the plant somatic cells via the Agrobacterium -mediated method. Suitable marker genes can help to quickly and efficiently screen many transformed materials and remove untransformed cells [ 8 ]. Conventional marker genes include the hpt gene, which encodes hygromycin phosphotransferase and confers resistance to hygromycin; the npt-II gene, which encodes neomycin phosphotransferase II and confers resistance to kanamycin, neomycin and geneticin; and the bar gene, which encodes phosphinothricin acetyltransferase and confers resistance to the herbicide phosphinothricin [ 23 , 51 , 52 ]. The type of resistance marker gene is determined in the vector, and hpt and npt-II are commonly used as marker genes for lily transformation [ 3 , 17 , 18 , 24 ]. Linking the GUS gene to a transformation vector for double or even triple marker gene screening combined with GUS histochemical staining is also an effective strategy for reducing the false positive rate of resistant plants [ 16 , 22 , 26 ] (Table  1 ).

Key parameters in the genetic transformation program

Ph of the medium.

The expression of the Agrobacterium virulence gene vir is the basis for the transformation of plant cells and the key to ensuring infection efficiency, which is strongly dependent on the pH of the medium [ 17 , 22 , 53 ]. Previous studies have shown that an acidic pH is more conducive to the expression of vir , and as the pH of the preculture or coculture medium decreases, the expression of vir is significantly upregulated [ 54 , 55 ]. Maintaining the pH at 5.2 effectively increased the genetic transformation efficiency of tomato cotyledons [ 56 ]. The virA and virG genes are switches that activate the expression of the vir gene [ 57 ]. Agrobacterium has a chemotactic system different from that of E. coli and is attracted to chemical inducers such as carbohydrates, amino acids and phenolic compounds [ 58 ]. High concentrations of chemical inducers bind to virA to induce the expression of the vir gene and trigger T-DNA transfer [ 59 ]. Acetylsyringone (AS) is a common class of natural phenolic compounds that can promote the direct entry of microorganisms into plant cells through wounds and achieve T-DNA transfer by activating the expression of the vir gene [ 58 , 60 , 61 , 62 ]. AS has been widely used in the genetic transformation of lily [ 23 , 27 ]. Although pH 7.0 is the most suitable environment for the growth of Agrobacterium , vir is more easily expressed under acidic conditions after the addition of AS, while vir expression is hardly induced under neutral pH conditions [ 54 , 63 ]. In a Lilium pumilum DC.Fisch. genetic transformation experiment, a stable pH of 5.8 in suspension and coculture media resulted in a somatic embryo transformation efficiency of 29.17% [ 26 ]. When the pH was adjusted to 5.0, the number of resistant calli increased significantly, and the transformation efficiency increased by 5.7–13% [ 17 ]. Ogaki et al. [ 23 ] reported that exogenous MES could effectively control the pH of the medium. This study further investigated the effect of adding different concentrations of MES (0, 10, 20, 50 and 100 mM) on the transformation efficiency of lily. The results showed that transient expression of the GUS gene could be observed only in coculture medium containing MES, and larger numbers of transgenic calli could be obtained by the addition of 10 mM MES buffer [ 30 ]. The above conclusions indicate that maintaining pH in the range of 5–6 values according to different varieties in the preculture and coculture stages is important for improving the efficiency of genetic transformation (Table  2 ).

Culture medium supplements

The composition of the medium is another rate-limiting factor affecting the genetic transformation efficiency of lily, and the process involves the preculture, coculture and regeneration of resistant plants. Many studies have shown that the addition or removal of certain compounds can significantly improve the efficiency of lily transformation (Table  2 ). MS medium, which contains 20.6 mM NH 4 NO 3 , is widely used in the tissue culture and genetic transformation of lily. However, the presence of NH 4 NO 3 limits the efficiency of genetic transformation in lily [ 23 , 26 , 64 ]. Previous studies have shown that virG transcription can be activated by low concentrations of phosphate [ 53 , 58 ]. When a low concentration of KH 2 PO 4 was used as a salt source instead of NH 4 NO 3 , there was no significant change in the number of regenerated resistant calli. In contrast, the complete removal of KH 2 PO 4 had a positive effect on lily transformation [ 22 ]. In a transformation study of the lily ‘Sorbonne’, it was found that the removal of KH 2 PO 4 , NH 4 NO 3 , KNO 3 or macroelements in the medium could significantly improve the transformation efficiency [ 28 ]. Montoro et al. [ 65 ] reported that Ga 2+ -free media significantly increased GUS activity in Brazilian rubber trees. However, another study concluded that the effect of GaCl 2 on plant transformation efficiency appears to be strongly dependent on genotype. Ga 2+ is considered one of the key factors involved in improving the efficiency of genetic transformation in improved lily genetic transformation systems. Increasing the GaCl 2 concentration from 0.44 g/L to 1.32 g/L significantly increased the germination coefficient of Lilium -resistant somatic embryos [ 17 ]. AS is an indispensable compound in the genetic transformation of lily, and its concentration is also a key factor; an AS concentration that is too high adversely affects T-DNA transfer [ 17 ]. Notably, researchers have found other compounds that can replace AS, and they can provide higher transformation efficiency. Chloroxynil (CX) is a class of phenolic compounds with a mode of action similar to that of AS that also improves the efficiency of treatment by activating the expression of vir. In the genetic transformation of lotus seeds, the transformation efficiency of explants treated with CX was 6 times greater than that of explants treated with AS [ 66 ]. Wei et al. [ 28 ] further confirmed the effect of CX in lily. When 4 μM CX was used, the transformation efficiency reached 11.1%, while 100 μM AS achieved only 6.6%, indicating that CX can replace AS in lily genetic transformation. Early studies showed that the promoting effect of carbohydrate substances other than glucose and xylose on vir gene activity was consistent with that of AS [ 67 , 68 ]. Further studies by Azadi et al. [ 22 ] showed that MS media supplemented with monosaccharides significantly inhibited the expression of the GUS gene, and no hygromycin-resistant lily calli were obtained. In contrast, adding sucrose significantly improved the efficiency of genetic transformation. In summary, for most series lilies, removing NH 4 NO 3 and adding an appropriate amount of AS has a positive effect on improving the genetic transformation efficiency. CX may be an excellent compound to replace AS, and it is worth further attempts in the future.

Bacterial concentration and infection time

The Agrobacterium concentration and infection time play pivotal roles in the transformation of lily. A low bacterial concentration and short infection duration will result in the failure of Agrobacterium to fully adhere to explant tissues, resulting in the inability to achieve effective transformation [ 69 ]. However, a high concentration of bacteria or long infection duration may also lead to rapid bacterial growth, which can cause severe damage to the recipient material [ 17 ]. Cell resistance varies greatly among different plant explants, and plant tolerance to different agrobacterium concentrations also differs. When the OD 600 was 0.8, the highest GUS expression rate was detected in embryogenic calli, but the percentage of resistant calli significantly decreased compared with that when the OD 600 was 0.6. For scales, the GUS expression rate and adventitious bud regeneration rate peaked when the OD 600 was 0.6. Infection time is also the key to determining transformation efficiency, and research shows that for embryonic calli and scales of Lilium pumilum, DC. Fisch, 15 min is the optimal time for Agrobacterium infection [ 26 ]. However, for embryogenic calli of the Oriental hybrid lily ‘Siberia’, an OD 600 of 0.4 was more beneficial for improving the transformation efficiency [ 18 ]. In addition, it has been reported that in the coculture process, the proliferation of Agrobacterium on the surface and around the callus increases with the removal of some elements, indicating that these elements have an inhibitory effect on Agrobacterium . Negative effects of bacterial overgrowth were observed when 10 mM MES was added to coculture media of sensitive varieties such as ‘Red Ruby’ and ‘Casa Blanca’. Therefore, screening different varieties of MES can effectively reduce bacterial growth and improve the transformation efficiency of lily [ 22 ].

Antibiotic selection

In the process of plant genetic transformation, an appropriate concentration of antibiotics can effectively inhibit the growth of non-transformed Cefatothin, and this is also a crucial step in determining the success of genetic transformation. Kanamycin, hygromycin and glyphosate have been used extensively for lily transformation due to their high availability and low toxicity, despite the occurrence of false positives in the screening of resistant plants [ 70 , 71 ] (Table  2 ). Lily explants of different genotypes have different antibiotic concentration requirements. Even within the same variety, different explant types have great differences in antibiotic tolerance [ 18 ]. Studies have shown that embryonic calli of Lilium pumilum DC. Fisch. The plants almost stopped growing and died after treatment with hygromycin supplemented at 40 mg·L −1 , resulting in extremely low growth and induction rates. However, a few scales of ‘White Heaven’ still formed complete buds under these conditions. Adjusting the concentration of hygromycin to 30 mg·L −1 reduced the browning rate of embryogenic calli by approximately 20% and significantly increased the growth and transformation rate. Therefore, 30 mg·L −1 and 40 mg·L −1 were the best hygromycin concentrations suitable for embryogenic calli and scales of Lilium pumilum DC. Fisch., respectively [ 26 ]. Another necessary antibiotic is a bacteriostatic antibiotic, which is mainly used to prevent the transformation of material from dying or difficult regeneration due to excessive Agrobacterium contamination. Cef is a common bacteriostatic agent used in plant transformation that has extensive resistance and inhibits the growth of Agrobacterium [ 72 , 73 ]. However, high concentrations of Cef can inhibit the growth of explant cells. Based on the results of studies on different lily varieties and explants, we believe that 300–400 mg·L −1 Cef may have a broad-spectrum effect [ 18 , 26 ]. The concentration of Agrobacterium may be a prerequisite for screening Cef concentrations. Even 400 mg·L −1 Cef had no bacteriostatic effect when the concentration of the bacterial solution was too high. When the OD600 of the bacterial solution is maintained within 0.2–0.4, 300 mg·L −1 Cef can have good antibacterial efficacy [ 18 ]. Therefore, it is necessary to combine the concentration of the bacterial solution with the concentration of the bacteriostatic agent during screening.

Preculture, infection and coculture procedures

Preculture, infection and coculture are the key steps in determining the success of plant genetic transformation. Previous studies have generally been conducted under the belief that the preculture of explants before transformation can effectively promote cell division so that they can maintain the best life state during infection and integrate foreign genes more easily [ 74 , 75 ]. The timing of preculture depends on the type and quality of the explants. Yan et al. [ 26 ] discussed the influence of preculture time on the transformation efficiency of L. pumilum and ‘White Heaven’. The results showed that the expression rate of GUS was lower in uncultured calli or scale explants. Similarly, compared with those of the control group, the proliferation and survival rates of the explant-treated group were significantly lower. For embryogenic calli, GUS expression and the proliferation rate were the highest in resistant calli after 10 days of preculture (66.67% and 63.33%, respectively). After 4 days of preculture, GUS expression and bud resistance began to decrease for the traumatized scales. Although the percentage of resistant buds was the highest after 2 days of preculture, a higher GUS expression rate appeared after 3 days (Table  2 ).

In lily, wounded explant materials are often more conducive to the transfer and integration of T-DNA, which can greatly improve the efficiency of genetic transformation [ 27 , 76 ]. Wei et al. [ 28 ] further confirmed this view. According to the results of Agrobacterium -mediated ‘Sobone’ genetic transformation, ultrasound treatment for 20 s can produce thousands of microwounds in explants, promote the penetration of Agrobacterium into the internal tissues of plants, and effectively improve the efficiency of transformation. The combination of heat shock and ultrasound had no significant effect. Coculture is an essential stage during which T-DNA is transferred into plant cells [ 77 ]. Therefore, coculture time is also an external factor that has been widely examined [ 75 ]. The time required for Agrobacterium -mediated gene transfer and integration into the plant genome varies widely depending on the genotype and explant type, usually ranging from a few hours to a few days [ 78 , 79 , 80 , 81 ]. Wu et al. [ 82 ] compared the transformation efficiency of bulb sections of Gladiolus under coculture for 3 days and 12 days, and the results showed that the transformation rate of coculture for 12 days was more than twice that for 3 days, indicating that a longer coculture time may benefit Agrobacterium infection and transformation. However, in Lilium pumilum DC. After more than 5 days of coculture, Fisch, which is also a typical bulbous flower, will cause severe browning and death of embryogenic calli and scales [ 26 ]. However, for the calli of the other two kinds of lilies, coculture for 7 days still maintained a high transformation efficiency, indicating that the tolerance of lily to Agrobacterium may depend on the genotype [ 15 , 27 ]. Drying plant tissue or cells before coculture can also promote T-DNA transfer [ 83 ]. The growth state and speed of calli in dry coculture were better than those in traditional media [ 84 ]. During subsequent resistance screening, only a few tissues were contaminated by the bacterial solution under dry conditions, and the regeneration rate of resistant calli increased significantly [ 34 ]. An appropriate low temperature during coculture also had a positive effect on T-DNA transfer [ 85 , 86 ]. The transformation efficiency of Boehmeria nivea (L.) Gaud. was significantly improved by coculture at 20 °C compared with 15 °C, 25 °C and 28 °C [ 87 ]. In the genetic transformation system of Gossypium hirsutum , 19 °C can significantly increase the regeneration rate of resistant calli and completely inhibit the proliferation of Agrobacterium . However, the effect of temperature on the genetic transformation efficiency of lily has not been clearly reported. In future studies, we can try to optimize the genetic transformation system for lily by adjusting the ambient temperature at each link.

Application of genetic transformation technology

Generation of the crispr/cas9 system.

With the continuous development of gene function research technology, gene modification has been widely used in basic plant research and molecular breeding [ 88 ]. Since CRISPR/Cas9 gene editing technology was successfully applied in Lotus japonicus , because of its simple design and limited operation, this technique has been successfully used in the study of flower anatomy and morphology, flower colour, flowering time, fragrance and stress resistance in various ornamental plants [ 89 , 90 , 91 , 92 ]. Yan et al. [ 26 ] established a stable and efficient genetic transformation system for two lily genotypes using somatic embryos and scales as explants and generated completely albino, light yellow and albino green chimeric mutants via directional knockout of the PDS gene; these authors successfully applied CRISPR/Cas9 technology to lily for the first time. The CRISPR/Cas9 system also validated the feasibility and efficiency of the two genetic transformation systems.

Application for improvement of plant morphogenesis

Morphogenetic genes are key factors that control plant organogenesis and somatic embryogenesis and determine the location of target cells to produce different structures or whole plants [ 8 ]. The functions of many morphogenetic genes have been identified in model plants and important cash crops and have been applied in scientific practice to increase the efficiency of regeneration and genetic transformation [ 93 , 94 ]. In lily, the somatic embryo has always been a good explant for genetic transformation. Plant somatic cells dedifferentiate into embryogenic stem cells under the action of external/internal genetic factors and then divide into somatic embryos. This process is the most critical stage for plant cells to become totipotent [ 95 ]. The most widely used method for somatic embryogenesis (SE) in various plants is the use of exogenous plant growth regulators, especially auxin [ 96 ]. Song et al. [ 17 ] reported that overexpression of LpABCB21 in lily could shorten the time required for SE without changing the exogenous PIC (Picloram). In contrast, the LpABCB21 mutant lines delayed somatic embryo generation by 1–3 days, but the induction rate of adventitious buds was significantly greater than that in the LpABCB21 -overexpressing lines. The study also indicated that the PILS (PIN-LIKES) family member LpPILS7 may participate in auxin regulation through the same mechanism as LpABCB21 , and the somatic embryo induction efficiency of the pils7 mutant was significantly reduced by approximately 10–60%. The importance of miRNAs in SE processes has also been validated in many dicot species and crops [ 82 , 97 ]. In Agrobacterium- mediated Lilium embryo transformation experiments, silencing lpu-miR171a and lpu-miR171b promoted starch accumulation and the expression of key cell cycle genes in calli, significantly accelerated the SE process in Lilium , and resulted in the same phenotype as overexpressing LpSCL6-II and LpSCL6-I . WUSCHEL is a typical gene family involved in the regulation of plant morphogenesis, and its expression is upregulated in many plant SE processes [ 98 , 99 ]. LlWOX9 and LlWOX11 reportedly play a positive regulatory role in the formation of bulbils by influencing cytokinin signalling [ 100 ]. However, the function of WUSCHEL members in Lilium embryogenesis remains to be further verified. It seems that changing the expression level of morphogenetic genes can be an effective means to improve the genetic transformation efficiency of lily, and this topic is worthy of further exploration in the future.

Genetic modification for agronomically important traits

At present, few studies have investigated genetic modification in lily, and most studies have validated the function of target genes only through transient gene transformation (Table  3 ). With the continuous improvement of the genetic transformation system for lily, a few key genes regulating important traits have been identified. In addition to influencing SE processes, many morphogenetic genes are involved in regulating plant organ formation or quality maintenance [ 101 , 102 , 103 ]. LaKNOX1 , a member of the homeobox gene family involved in regulating plant organogenesis, was also further validated in 'Siberia' and 'Sorbonne' [ 18 ]. A recent study revealed that a key gene, LdXERICO , is involved in the regulation of dormancy in Lilium davidii var. unicolour , which indicated that the maintenance of dormancy depends on the ABA-related pathway and that the transcription of LdXERICO is inhibited by the temperature response factor LdICE1 during low-temperature storage, which eventually leads to lily sprouting [ 3 ]. Recently, the LoNFYA7-LoVIL1 module has also been shown to play a key role in orchestrating the phase transition from slow to fast growth in lily bulbs [ 104 ]. Biological and abiotic stresses have a great impact on plant growth and development, and these stresses usually disrupt cellular mechanisms by inducing changes at the physiological, biochemical, and molecular levels in plants [ 105 ]. The identification of key genes involved in the regulation of the stress response in lily was aimed at improving plant resistance to biotic and abiotic stresses. Low temperature, drought, salt stress and abscisic acid treatment can significantly upregulate the expression of LlNAC2 , a member of the NAC transcription factor family. Overexpression of the LlNAC2 gene in tobacco significantly enhances the tolerance of transgenic plants to various abiotic stresses [ 106 ]. Chen et al. [ 18 ] used the genetic transformation system of lily to transform LlNAC2 and successfully generated a transgenic line, which provided favourable support for further clarifying the function of LlNAC2 in coping with abiotic stress in lily species. Typical biological stresses, including bacteria, fungi, viruses, insects and other diseases and pests, seriously negatively affect the quality of ornamental plants [ 89 ]. Several researchers have attempted to increase the resistance of lily plants to pathogens or pests by increasing or decreasing the expression of certain genes. Pratylenchus penetrans (RLN) is one of the main pests and diseases encountered in lily production. The overexpression of the rice cystatin (Oc-IΔD86 ) gene in Lilium longiflorum cv. 'Nellie White' showed that the resistance of transgenic lily to RLN infection was significantly enhanced, and the total nematode population decreased by 75 ± 5%. Compared with wild-type plants, OcIΔD86 -overexpressing plants also exhibited improved growth and development [ 107 ]. Plant resistance to viruses is usually established by transferring the coat protein-encoding gene of the virus into the plant [ 89 ]. Azadi et al. [ 22 ] introduced a cucumber mosaic virus (CMV) replicase defective gene ( CMV2-GDD ) into lily using an Agrobacterium -mediated genetic transformation system and identified two transgenic strains that showed stronger resistance to CMV. In Lilium oriental cv. 'Star Gazer', overexpression of the rice chitinase 10 ( RCH10 ) gene enhanced the resistance of lily to Botrytis elliptica [ 19 ]. Du et al. [ 108 ] identified a gene named LhSorPR4-2 , which encodes a disease course-related protein involved in fighting Botrytis elliptica infection in lily, and the overexpression of LhSorPR4-2 significantly enhanced the resistance of lily to Botrytis . This study also revealed that the function of LhSorPR4-2 was closely related to its chitinase activity. Another study showed that the transcription level of the resistance gene LrPR10-5 was significantly increased in transgenic ‘Siberia’ plants that overexpressed LrWRKY1 , which subsequently promoted resistance to F. oxysporum [ 109 ].

Conclusions

Since the first successful transformation event in lily, remarkable progress has been made; a variety of lily genetic transformation systems have been gradually established, and many excellent new germplasms have been obtained. However, the genetic transformation of lily still faces great challenges due to its strong genotypic dependence. Most related studies have focused on optimizing existing systems, and applicable genetic transformation systems have not yet been established for most lily strains with high market value. For a long time, how to stably and efficiently deliver recombinant gene vectors into plant cells has been the focus of most scholars. At present, the most common delivery method is Agrobacterium -mediated transformation. In addition to the cumbersome tissue culture process, the transformation efficiency also depends greatly on the genotype. The choice of explants for DNA, strains and vectors; culture conditions; and effective selection markers are all major factors that play pivotal roles in successful transformation. At present, most transgenic work in lily is limited by laboratory-scale gene function verification, and even after successful transformation, it is not easy to obtain stable transgenic plants. In recent years, various types of Rhizobium , including Ensifer adhaerens , Ochrobactrum haywardense and Sinorhizobium meliloti , have shown great potential in the transformation of nonagricultural bacterial systems. Some studies have revealed an invisible mechanism for delivering DNA into plant cells, where Sinorhizobium meliloti can infect both monocotyledonous and dicotyledonous plants [ 164 ]. Another way to improve the traditional transformation model is to coexpress developmental regulatory factors or morphogenetic genes during transformation. The overexpression of the developmental regulatory factors GROWTH-REGULATING factor (GRF) and Boby room (Bbm) in maize and sorghum, for which it is difficult to achieve genetic transformation, can significantly improve transformation efficiency [ 101 , 165 ]. Pollen tube transformation is a transformation system that does not require tissue culture, but this method is suitable only for model plants such as Arabidopsis and a few closely related plants. The pollen magnetic transfection-mediated transformation method can be applied to lily, but it may be species- or varietal specific. Zhang et al. [ 166 ] optimized pollen culture conditions, established a new method for the transient transformation of pollen magnetic beads, and concluded that the transformation efficiency was positively correlated with the transverse diameter of pollen and negatively correlated with the ratio of longitudinal diameter to transverse diameter. This study also evaluated the transformation efficiency of Lilium regale L. ‘Sweet Surrender’ and Lilium leucanthum ; L. ‘Sweet Surrender’ and Lilium leucanthum reached 85.80% and 54.47%, respectively, but successful transformation was not achieved in Lilium davidii var. unicolour . Particle bombardment and the electrical shock method are also common methods used in plant genetic transformation. At present, the main explants of the electroshock method are plant protoplasts, but because of their high cost, abundance of chimaeras after transformation and limited stable expression in offspring, these methods cannot be applied to large-scale lily plants. Therefore, exploring genetic transformation systems based on non-tissue culture methods is expected to alleviate the pressure of genetic transformation of lily in the future. Cao et al. [ 167 ] reported a cut-dip-bud (CDB) delivery system. Briefly, the CDB delivery system consists of cutting the junction of plant roots under nonsterile conditions, infecting the upper end with Agrobacterium , taking positive new roots after culture, cutting them into segments and culturing them again to obtain regenerated and transformed plants. Researchers have studied the effects on rubber grass ( Taraxacum koko-saghyz Rodin , TKS), Ipomoea batatas [L.] Lam.), Ailanthus altissima (Mill) Swingle, and Aralia elata (Miq.) The CDB system has been tested in several difficult-to-transform plants, including three woody plants, one of which is Clerodendrum chinense Mabb. The results showed that the CDB delivery system has wide applicability in plant genetic transformation. Furthermore, there is a great need for the validation of promoters other than CaMV35S to achieve optimal expression of transforming genes [ 8 ].

Notably, various plant genetic transformation systems have been further applied to establish RNA interference (RNAi) and gene editing technology systems. To date, there have been few reports on the application of RNAi and CRISPR/Cas9-based gene editing techniques in lilies. In 2019, Yan established a stable and efficient genetic transformation system for somatic embryo regeneration for the first time and successfully conducted targeted gene editing based on CRISPR/Cas9 [ 20 , 26 ]. As one of the sharpest tools in genetic technology, CRISPR/Cas9 “gene scissors” have set off a research boom in basic plant research and directed breeding work. It has great application potential for improving yield, quality, herbicide resistance, abiotic stress resistance and disease resistance. However, there are few successful cases of gene editing using CRISPR/Cas9 technology in lily, which may be related to its high heterozygosity. The stable genetic transformation system has not been widely used, which leads to many difficulties in the study of gene function. With the continuous improvement of the technical system of lily genetic transformation and the emergence of new delivery methods, the combination of multiple transformations may be the only way to develop functional lily genomics in the future, and major breakthroughs in genetic engineering applications in lily breeding are expected not to occur.

Availability of data and materials

The datasets used and/or analysed during the current study available from the corresponding author on reasonable request. No datasets were generated or analysed during the current study.

Abbreviations

Embryogenesis

Acetosyringone

REGULATING factor

Cut-dip-bud

RNA interference

Liu X, Gu J, Wang J, Lu Y. Lily breeding by using molecular tools and transformation systems. Mol Biol Rep. 2014;41:6899–908.

Article   PubMed   CAS   Google Scholar  

Park S, Han N, Lee J, Lee JN, An S, Bae S. Anti-melanogenic effects of Lilium lancifolium root extract via downregulation of PKA/CREB and MAPK/CREB signaling pathways in B16F10 cells. Plant (Basel). 2023;12:3666.

CAS   Google Scholar  

Fan X, Zou X, Fu L, Yang Y, Li M, Wang C, et al. The RING-H2 gene LdXERICO plays a negative role in dormancy release regulated by low temperature in Lilium davidii var. unicolor . Hortic Res. 2023. https://doi.org/10.1093/hr/uhad030 .

Article   PubMed   PubMed Central   Google Scholar  

Li JW, Zhang XC, Wang MR, Bi WL, Faisal M, Da Silva JAT, et al. Development, progress and future prospects in cryobiotechnology of Lilium spp. Plant Method. 2019;15:125.

Article   Google Scholar  

Chen DF. Genetic transformation of wheat. Method Mol Biol. 1998;81:373–82.

Dayal S, Lavanya M, Devi P, Sharma KK. An efficient protocol for shoot regeneration and genetic transformation of pigeonpea [ Cajanus cajan (L) Millsp] using leaf explants. Plant Cell Rep. 2003;21:1072–9.

Ramkumar TR, Lenka SK, Arya SS, Bansal KC. A short history and perspectives on plant genetic transformation. Methods Mol Biol. 2020;2124:39–68.

Choudhury A, Rajam MV. Genetic transformation of legumes: an update. Plant Cell Rep. 2021;40:1813–30.

Zambryski P, Joos H, Genetello C, Leemans J, Montagu MV, Schell J. Ti plasmid vector for the introduction of DNA into plant cells without alteration of their normal regeneration capacity. EMBO J. 1983;2:2143–50.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Cohen A, Meredith CP. Agrobacterium -mediated transformation of Lilium . Acta Hortic. 1992;325:611–8.

Hensel G. Genetic transformation of Triticeae cereals —summary of almost three-decade’s development. Biotechnol Adv. 2020;40: 107484.

Article   PubMed   Google Scholar  

Chilton MD, Drummond MH, Merio DJ, Sciaky D, Montoya AL, Gordon MP, et al. Stable incorporation of plasmid DNA into higher plant cells: the molecular basis of crown gall tumorigenesis. Cell. 1977;11:263–71.

Krenek P, Samajova O, Luptovciak I, Doskocilova A, Komis G, Samaj J. Transient plant transformation mediated by Agrobacterium tumefaciens : principles, methods and applications. Biotechnol Adv. 2015;33:1024–42.

Gelvin SB. Agrobacterium -mediated plant transformation: the biology behind the “gene-jockeying” tool. Microbiol Mol Biol Rev. 2003;67:16–37.

Hoshi Y, Kondo M, Kobayashi H, Mori S, Nakano M. Agrobacterium -mediated transformation of Lilium longiflorum . Acta Hortic. 2005;673:543–7.

Article   CAS   Google Scholar  

Qi Y, Du L, Quan Y, Tian F, Liu Y, Wang Y. Agrobacterium -mediated transformation of embryogenic cell suspension cultures and plant regeneration in Lilium tenuifolium oriental × trumpet ‘Robina.’ Acta Physiol Plant. 2014;36:2047–57.

Song S, Yan R, Wang C, Wang J, Sun H. Improvement of a genetic transformation system and preliminary study on the function of LpABCB21 and LpPILS7 based on somatic embryogenesis in Lilium pumilum DC. Fisch Int J Mol Sci. 2020;21:6784.

Chen Y, Hou X, Zheng Y, Lyu Y. The establishment of a genetic transformation system and the acquisition of transgenic plants of oriental hybrid lily ( Lilium L.). Int J Mol Sci. 2023;24:782.

González FFNDC, Davey MR, Cancho Sanchez E, Wilson ZA. Conferred resistance to Botrytis cinerea in Lilium by overexpression of the RCH10 chitinase gene. Plant Cell Rep. 2015;34:1201–9.

Yan R, Song S, Li H, Sun H. Functional analysis of the eTM-miR171-SCL6 module regulating somatic embryogenesis in Lilium pumilum DC. Fisch. Hortic Res. 2022. https://doi.org/10.1093/hr/uhac045 .

Azadi P, Bagheri H, Nalousi AM, Nazari F, Chandler SF. Current status and biotechnological advances in genetic engineering of ornamental plants. Biotechnol Adv. 2016;34:1073–90.

Azadi P, Otang NV, Supaporn H, Khan RS, Chin DP, Nakamura I, et al. Increased resistance to cucumber mosaic virus (CMV) in Lilium transformed with a defective CMV replicase gene. Biotechnol Lett. 2011;33:1249–55.

Ogaki M, Furuichi Y, Kuroda K, Chin DP, Ogawa Y, Mii M. Importance of co-cultivation medium pH for successful Agrobacterium -mediated transformation of Lilium x formolongi . Plant Cell Rep. 2008;27:699–705.

Liu J, Zhang J, Xu B, Jia C, Zhang J, Tan G, et al. Regeneration and production of transgenic Lilium longiflorum via Agrobacterium tumefaciens . In Vitro Cell Dev Biol Plant. 2011;47:348–56.

Mercuri A, De Benedetti L, Bruna S, Bregliano R, Bianchini C, Foglia G, et al. Agrobacterium -mediated transformation with rol genes of Lilium longiflorum thunb. Acta Hortic. 2003;612:129–36.

Yan R, Wang Z, Ren Y, Li H, Liu N, Sun H. Establishment of efficient genetic transformation systems and application of CRISPR/Cas9 genome editing technology in Lilium pumilum DC. Fisch. and Lilium longiflorum white heaven. Int J Mol Sci. 2019;20:2920.

Hoshi Y, Kondo M, Mori S, Adachi Y, Nakano M, Kobayashi H. Production of transgenic lily plants by Agrobacterium -mediated transformation. Plant Cell Rep. 2004;22:359–64.

Wei C, Cui Q, Lin Y, Jia G. Efficient Agrobacterium -mediated transformation of Lilium oriental “Sorbonne” with genes encoding anthocyanin regulators. Can J Plant Sci. 2017;97:796–807.

Li QH, Hong B, Tong Z, Ma C, Guan AN, Yu JJ, et al. Establishment of regeneration system and transformation of Zm401 gene in Lilium longiflorum × L. formosanum . Chin J Agric Biotechnol. 2008;5:113–9.

Abbasi H, Naderi R, Kafi M, Azadi P, Shakh-Asadi M, Okazaki K. Effect of ‘Chloroxynil’ on Agrobacterium -mediated transformation efficiency of Lilium cv ‘Manissa.’ Sci Hortic. 2020;271: 109404.

Fu L, Zhang J, Li M, Wang C, Chen Y, Fan X, et al. ldi-miR396- LdPMaT1 enhances reactive oxygen species scavenging capacity and promotes drought tolerance in Lilium distichum Nakai autotetraploids. Plant Cell Environ. 2023. https://doi.org/10.1111/pce.14783 .

Chakraborty N, Chakraborty P, Sen M, Bandopadhyay R. Choice of explant for plant genetic transformation. In: Rustgi S, Luo H, editors. Biolistic DNA delivery in plants: methods and protocols. New York: Springer; 2020. p. 107–23.

Chapter   Google Scholar  

Mori S, Adachi Y, Horimoto S, Suzuki S, Nakano M. Callus formation and plant regeneration in various Lilium species and cultivars. In Vitro Cell Dev Biol Plant. 2005;41:783–8.

Wang Y, Van Kronenburg B, Menzel T, Maliepaard C, Shen X, Krens F. Regeneration and Agrobacterium -mediated transformation of multiple lily cultivars. Plant Cell Tissue Organ Cult. 2012;111:113–22.

Potrykus I. Gene transfer to plants: assessment of published approaches and results. Annu Rev Plant Biol. 1991;42:205–25.

Cohen A, Lipsky A, Arazi T, Ion A, Stav R, Sandler-Ziv D, et al. Efficient genetic transformation of Lilium longiflorum and Ornithogalum dubium by particle acceleration followed by prolonged selection in liquid medium. Acta Hortic. 2004;651:131–8.

Mookkan M, Nelson-Vasilchik K, Hague J, Zhang ZJ, Kausch AP. Selectable marker independent transformation of recalcitrant maize inbred B73 and sorghum P898012 mediated by morphogenic regulators baby boom and WUSCHEL2. Plant Cell Rep. 2017;36:1477–91.

Nyaboga EN, Njiru JM, Tripathi L. Factors influencing somatic embryogenesis, regeneration, and Agrobacterium -mediated transformation of cassava ( Manihot esculenta Crantz) cultivar TME14. Front Plant Sci. 2015;6:411.

Su H, Jiao YT, Wang FF, Liu YE, Niu WL, Liu GT, et al. Overexpression of VpPR10.1 by an efficient transformation method enhances downy mildew resistance in V. vinifera . Plant Cell Rep. 2018;37:819–32.

Do PT, Lee H, Nelson-Vasilchik K, Kausch A, Zhang ZJ. Rapid and efficient genetic transformation of sorghum via Agrobacterium -mediated method. Curr Protoc Plant Biol. 2018;3: e20077.

Palomo-Ríos E, Narváez I, Pliego-Alfaro F, Mercado JA. Olive ( Olea europaea L. ) genetic transformation: current status and future prospects. Genes (Basel). 2021;12:386.

Raghuwanshi A, Birch RG. Genetic transformation of sweet sorghum. Plant Cell Rep. 2010;29:997–1005.

Tzfira T, Citovsky V. Agrobacterium -mediated genetic transformation of plants: biology and biotechnology. Curr Opin Biotechnol. 2006;17:147–54.

Zhang B. Agrobacterium -mediated genetic transformation of cotton. Method Mol Biol. 2019;1902:19–33.

Xi Y, Ge Y, Wang ZY. Genetic transformation of switchgrass. Method Mol Biol. 2009;581:53–9.

Tichá M, Illésová P, Hrbáčková M, Basheer J, Novák D, Hlaváčková K, et al. Tissue culture, genetic transformation, interaction with beneficial microbes, and modern bio-imaging techniques in alfalfa research. Crit Rev Biotechnol. 2020;40:1265–80.

Rosellini D, Capomaccio S, Ferradini N, Sardaro MLS, Nicolia A, Veronesi F. Non-antibiotic, efficient selection for alfalfa genetic engineering. Plant Cell Rep. 2007;26:1035–44.

Gao N, Shen W, Cao Y, Su Y, Shi W. Influence of bacterial density during preculture on Agrobacterium -mediated transformation of tomato. Plant Cell Tissue Organ Cult. 2009;98:321–30.

Barik DP, Mohapatra U, Chand PK. Transgenic grasspea ( Lathyrus sativus L.): factors influencing Agrobacterium -mediated transformation and regeneration. Plant Cell Rep. 2005;24:523–31.

Krishna G, Reddy PS, Ramteke PW, Bhattacharya PS. Progress of tissue culture and genetic transformation research in pigeon pea [ Cajanus cajan (L.) Millsp]. Plant Cell Rep. 2010;29:1079–95.

Ganguly S, Ghosh G, Ghosh S, Purohit A, Chaudhuri RK, Das S, et al. Plumular meristem transformation system for chickpea: an efficient method to overcome recalcitrant tissue culture responses. Plant Cell Tissue Organ Cult. 2020;142:493–504.

Ghosh G, Purohit A, Ganguly S, Chaudhuri RK, Chakraborti D. In vitro shoot grafting on rootstock: an effective tool for Agrobacterium-mediated transformation of pigeonpea ( Cajanus cajan (L.) Millsp.). Plant Biotechnol. 2014;31:301–8.

Winans SC. Transcriptional induction of an Agrobacterium regulatory gene at tandem promoters by plant-released phenolic compounds, phosphate starvation, and acidic growth media. J Bacteriol. 1990;172:2433–8.

Godwin I, Todd G, Ford-Lloyd B, Newbury HJ. The effects of acetosyringone and pH on Agrobacterium -mediated transformation vary according to plant species. Plant Cell Rep. 1991;9:671–5.

Vernade D, Herrera-Estrella A, Wang K, Van Montagu M. Glycine betaine allows enhanced induction of the Agrobacterium tumefaciens vir genes by acetosyringone at low pH. J Bacteriol. 1988;170:5822–9.

Wu YF, Chen Y, Liang XM, Wang XZ. An experimental assessment of the factors influencing Agrobacterium -mediated transformation in tomato. Russ J Plant Physiol. 2006;53:252–6.

Palmer AG, Gao R, Maresh J, Erbil WK, Lynn DG. Chemical biology of multi-host/pathogen interactions: chemical perception and metabolic complementation. Annu Rev Phytopathol. 2004;42:439–64.

McCullen CA, Binns AN. Agrobacterium tumefaciens and plant cell interactions and activities required for interkingdom macromolecular transfer. Annu Rev Cell Dev Biol. 2006;22:101–27.

Guo M, Huang Z, Yang J. Is there any crosstalk between the chemotaxis and virulence induction signaling in Agrobacterium tumefaciens ? Biotechnol Adv. 2017;35:505–11.

Song YN, Shibuya M, Ebizuka Y, Sankawa U. Synergistic action of phenolic signal compounds and carbohydrates in the induction of virulence gene expression of Agrobacterium tumefaciens . Chem Pharm Bull (Tokyo). 1991;39:2613–6.

Stachel SE, Nester EW, Zambryski PC. A plant cell factor induces Agrobacterium tumefaciens vir gene expression. Proc Natl Acad Sci U S A. 1986;83:379–83.

Verma AK, Chand L. Agrobacterium-mediated transformation of pigeonpea ( Cajanus cajan L.) with uidA and cryIA(b) genes. Physiol Mol Biol Plants. 2005;11:99–109.

Li L, Jia Y, Hou Q, Charles TC, Nester EW, Pan SQ. A global pH sensor: Agrobacterium sensor protein ChvG regulates acid-inducible genes on its two chromosomes and Ti plasmid. Proc Natl Acad Sci U S A. 2002;99:12369–74.

Azadi P, Ntui V, Chin D, Mii M. Genetic transformation and metabolic engineering of Lilium . In: Ramawat KG, Merillon JM, editors. Bulbous plants. Boca Raton: CRC Press; 2016. p. 197–213.

Google Scholar  

Montoro P, Teinseree N, Rattana W, Kongsawadworakul P, Michaux-Ferriere N. Effect of exogenous calcium on Agrobacterium tumefaciens -mediated gene transfer in Hevea brasiliensis (rubber tree) friable calli. Plant Cell Rep. 2000;19:851–5.

Kimura M, Cutler S, Isobe S. A novel phenolic compound, chloroxynil, improves Agrobacterium -mediated transient transformation in Lotus japonicus . PLoS ONE. 2015;10: e0131626.

Ahn JH, Lee JS. Sugar acts as a regulatory signal on the wound-inducible expression of SbHRGP3 ::GUS in transgenic plants. Plant Cell Rep. 2003;22:286–93.

Wise AA, Voinov L, Binns AN. Intersubunit complementation of sugar signal transduction in VirA heterodimers and posttranslational regulation of VirA activity in Agrobacterium tumefaciens . J Bacteriol. 2005;187:213–23.

Pushyami B, Beena MR, Sinha MK, Kirti PB. In vitro regeneration and optimization of conditions for Agrobacterium mediated transformation in jute. Corchorus capsularis J Plant Biochem Biotechnol. 2011;20:39–46.

Nakano M, Otani M. Plant regeneration and Agrobacterium -mediated genetic transformation systems in liliaceous ornamental plants. Plant Biotechnol (Tokyo). 2020;37:129–40.

Suzuki S, Oota M, Nakano M. Embryogenic callus induction from leaf explants of the liliaceous ornamental plant, Agapanthus praecox ssp. orientalis (Leighton) Leighton: histological study and response to selective agents. Sci Hortic. 2002;95:123–32.

Ahmed RI, Ding A, Xie M, Kong Y. Progress in optimization of Agrobacterium -mediated transformation in Sorghum ( Sorghum bicolor ). Int J Mol Sci. 2018;19:2983.

Sunilkumar G, Rathore KS. Transgenic cotton: factors influencing Agrobacterium -mediated transformation and regeneration. Mol Breed. 2001;8:37–52.

Lin YJ, Zhang Q. Optimising the tissue culture conditions for high efficiency transformation of indica rice. Plant Cell Rep. 2005;23:540–7.

Toki S, Hara N, Ono K, Onodera H, Tagiri A, Oka S, et al. Early infection of scutellum tissue with Agrobacterium allows high-speed transformation of rice. Plant J. 2006;47:969–76.

Liu Z, Park BJ, Kanno A, Kameya T. The novel use of a combination of sonication and vacuum infiltration in Agrobacterium -mediated transformation of kidney bean ( Phaseolus vulgaris L.) with lea Gene. Mol Breed. 2005;16:189–97.

Liu SJ, Wei ZM, Huang JQ. The effect of co-cultivation and selection parameters on Agrobacterium -mediated transformation of Chinese soybean varieties. Plant Cell Rep. 2008;27:489–98.

Chen X, Equi R, Baxter H, Berk K, Han J, Agarwal S, et al. A high-throughput transient gene expression system for switchgrass ( Panicum virgatum L.) seedlings. Biotechnol Biofuel. 2010;3:9.

Sun HJ, Uchii S, Watanabe S, Ezura H. A highly efficient transformation protocol for Micro-Tom, a model cultivar for tomato functional genomics. Plant Cell Physiol. 2006;47:426–31.

Vergne P, Maene M, Gabant G, Chauvet A, Debener T, Bendahmane M. Somatic embryogenesis and transformation of the diploid Rosa chinensis cv Old Blush. Plant Cell Tissue Organ Cult. 2010;100:73–81.

Zhao J, Li ZT, Cui J, Henny RJ, Gray DJ, Xie J, et al. Efficient somatic embryogenesis and Agrobacterium -mediated transformation of pothos ( Epipremnum aureum ) ‘Jade.’ Plant Cell Tissue Organ Cult. 2013;114:237–47.

Wu XM, Kou SJ, Liu YL, Fang YN, Xu Q, Guo WW. Genomewide analysis of small RNAs in nonembryogenic and embryogenic tissues of citrus: microRNA- and siRNA-mediated transcript cleavage involved in somatic embryogenesis. Plant Biotechnol J. 2015;13:383–94.

Cheng M, Hu T, Layton J, Liu CN, Fry JE. Desiccation of plant tissues post- Agrobacterium infection enhances T-DNA delivery and increases stable transformation efficiency in wheat. In Vitro Cell Dev Biol Plant. 2003;39:595–604.

Vogel J, Hill T. High-efficiency Agrobacterium -mediated transformation of Brachypodium distachyon inbred line Bd21-3. Plant Cell Rep. 2008;27:471–8.

Dillen W, De Clercq J, Kapila J, Zambre M, Van Montagu M, Angenon G. The effect of temperature on Agrobacterium tumefaciens -mediated gene transfer to plants. Plant J. 1997;12:1459–63.

Fullner KJ, Nester EW. Temperature affects the T-DNA transfer machinery of Agrobacterium tumefaciens . J Bacteriol. 1996;178:1498–504.

An X, Wang B, Liu L, Jiang H, Chen J, Ye S, et al. Agrobacterium -mediated genetic transformation and regeneration of transgenic plants using leaf midribs as explants in ramie [ Boehmeria nivea (L.) Gaud]. Mol Biol Rep. 2014;41:3257–69.

Zhang Y, Ma X, Xie X, Liu YG. CRISPR/Cas9-based genome editing in plants. In: Weeks DP, Yang B, editors. Progress in molecular biology and translational science. Cambridge: Academic Press; 2017. p. 133–50.

Boutigny AL, Dohin N, Pornin D, Rolland M. Overview and detectability of the genetic modifications in ornamental plants. Hortic Res. 2020;7:11.

Kishi-Kaboshi M, Aida R, Sasaki K. Generation of gene-edited Chrysanthemum morifolium using multicopy transgenes as targets and markers. Plant Cell Physiol. 2017;58:216–26.

PubMed   CAS   Google Scholar  

Wang C, Li Y, Wang N, Yu Q, Li Y, Gao J, et al. An efficient CRISPR/Cas9 platform for targeted genome editing in rose ( Rosa hybrida ). J Integr Plant Biol. 2023;65:895–9.

Wang L, Wang L, Tan Q, Fan Q, Zhu H, Hong Z, et al. Efficient inactivation of symbiotic nitrogen fixation related genes in Lotus japonicus using CRISPR-Cas9. Front Plant Sci. 2016;7:1333.

PubMed   PubMed Central   Google Scholar  

Kausch AP, Nelson-Vasilchik K, Hague J, Mookkan M, Quemada H, Dellaporta S, et al. Edit at will: genotype independent plant transformation in the era of advanced genomics and genome editing. Plant Sci. 2019;281:186–205.

Lowe K, La Rota M, Hoerster G, Hastings C, Wang N, Chamberlin M, et al. Rapid genotype “independent” Zea mays L. (maize) transformation via direct somatic embryogenesis. In Vitro Cell Dev Biol Plant. 2018;54:240–52.

Nowak K, Wójcikowska B, Gaj MD. ERF022 impacts the induction of somatic embryogenesis in Arabidopsis through the ethylene-related pathway. Planta. 2015;241:967–85.

Jiménez VM. Involvement of plant hormones and plant growth regulators on in vitro somatic embryogenesis. Plant Growth Regul. 2005;47:91–110.

Long JM, Liu CY, Feng MQ, Liu Y, Wu XM, Guo WW. miR156-SPL modules regulate induction of somatic embryogenesis in citrus callus. J Exp Bot. 2018;69:2979–93.

Jha P, Ochatt SJ, Kumar V. WUSCHEL : a master regulator in plant growth signaling. Plant Cell Rep. 2020;39:431–44.

Zuo J, Niu QW, Frugis G, Chua NH. The WUSCHEL gene promotes vegetative-to-embryonic transition in Arabidopsis . Plant J. 2002;30:349–59.

He G, Cao Y, Wang J, Song M, Bi M, Tang Y, et al. WUSCHEL -related homeobox genes cooperate with cytokinin to promote bulbil formation in Lilium lancifolium . Plant Physiol. 2022;190:387–402.

Lowe K, Wu E, Wang N, Hoerster G, Hastings C, Cho MJ, et al. Morphogenic regulators baby boom and WUSCHEL improve monocot transformation. Plant Cell. 2016;28:1998–2015.

Sun L, Song SL, Yang Y, Sun HM. Melatonin regulates lily bulblet development through the LoBPM3-LoRAV module. Ornam Plant Res. 2022;2:1–11.

Sun Q, Zhang B, Yang C, Wang W, Xiang L, Wang Y, et al. Jasmonic acid biosynthetic genes TgLOX4 and TgLOX5 are involved in daughter bulb development in tulip ( Tulipa gesneriana ). Hortic Res. 2022. https://doi.org/10.1093/hr/uhac006 .

Pan W, Li J, Du Y, Zhao Y, Xin Y, Wang S, et al. Epigenetic silencing of callose synthase by VIL1 promotes bud-growth transition in lily bulbs. Nat Plant. 2023;9:1451–67.

Parmar N, Singh KH, Sharma D, Singh L, Kumar P, Nanjundan J, et al. Genetic engineering strategies for biotic and abiotic stress tolerance and quality enhancement in horticultural crops: a comprehensive review. 3 Biotech. 2017;7:239.

Yong Y, Zhang Y, Lyu Y. A stress-responsive NAC transcription factor from tiger lily (LlNAC2) interacts with LlDREB1 and LlZHFD4 and enhances various abiotic stress tolerance in Arabidopsis . Int J Mol Sci. 2019;20:3225.

Vieira P, Wantoch S, Lilley CJ, Chitwood DJ, Atkinson HJ, Kamo K. Expression of a cystatin transgene can confer resistance to root lesion nematodes in Lilium longiflorum cv ‘Nellie White.’ Transgenic Res. 2015;24:421–32.

Du W, Chai N, Sun Z, Wang H, Liu S, Sui S, et al. Full-length transcriptome characterization and functional analysis of pathogenesis-related proteins in Lilium oriental hybrid ‘Sorbonne’ infected with Botrytis elliptica . Int J Mol Sci. 2022;24:425.

Li S, Liu G, Pu L, Liu X, Wang Z, Zhao Q, et al. WRKY transcription factors actively respond to Fusarium oxysporum in Lilium regale . Phytopathology. 2021;111:1625–37.

Tong Z, Li Q, Yang Y, Dai F, Gao J, Hong B. Isolation and expression analysis of LoPIP2 , a lily ( Lilium oriental hybrids) aquaporin gene involved in desiccation-induced anther dehiscence. Sci Hortic. 2013;164:316–22.

Wu Z, Liang J, Wang C, Zhao X, Zhong X, Cao X, et al. Overexpression of lily HsfA3s in Arabidopsis confers increased thermotolerance and salt sensitivity via alterations in proline catabolism. J Exp Bot. 2018;69:2005–21.

Wu Z, Liang J, Zhang S, Zhang B, Zhao Q, Li G, et al. A canonical DREB2-type transcription factor in lily is post-translationally regulated and mediates heat stress response. Front Plant Sci. 2018;9:243.

Zhang T, Sun M, Guo Y, Shi X, Yang Y, Chen J, et al. Overexpression of LiDXS and LiDXR from lily ( Lilium ’Siberia’) enhances the terpenoid content in tobacco flowers. Front Plant Sci. 2018;9:909.

Wu Z, Liang J, Wang C, Ding L, Zhao X, Cao X, et al. Alternative splicing provides a mechanism to regulate LlHSFA3 function in response to heat stress in lily. Plant Physiol. 2019;181:1651–67.

Tasaki K, Yamagishi M, Masuta C. Virus-induced gene silencing in lilies using cucumber mosaic virus vectors. Methods Mol Biol. 2020;2172:1–13.

Fu Y, Zhu Y, Yang W, Xu W, Li Q, Chen M, et al. Isolation and functional identification of a Botrytis cinerea -responsive caffeoyl-CoA O-methyltransferase gene from Lilium regale wilson. Plant Physiol Biochem. 2020;157:379–89.

Zhang T, Guo Y, Shi X, Yang Y, Chen J, Zhang Q, et al. Overexpression of LiTPS2 from a cultivar of lily ( Lilium ’Siberia’) enhances the monoterpenoids content in tobacco flowers. Plant Physiol Biochem. 2020;151:391–9.

Jiang X, Chi X, Zhou R, Li Y, Li W, Liu Q, et al. Transcriptome profiling to identify tepal cell enlargement and pigmentation genes and the function of LtEXLB1 in Lilium tsingtauense . Funct Plant Biol. 2021;48:241–56.

Yin X, Zhang Y, Zhang L, Wang B, Zhao Y, Irfan M, et al. Regulation of MYB transcription factors of anthocyanin synthesis in lily flowers. Front Plant Sci. 2021;12: 761668.

Abbas F, Nian X, Zhou Y, Ke Y, Liu L, Yu R, et al. Putative regulatory role of hexokinase and fructokinase in terpenoid aroma biosynthesis in Lilium “Siberia.” Plant Physiol Biochem. 2021;167:619–29.

Ding L, Wu Z, Teng R, Xu S, Cao X, Yuan G, et al. LlWRKY39 is involved in thermotolerance by activating LlMBF1c and interacting with LlCaM3 in lily ( Lilium longiflorum ). Hortic Res. 2021;8:36.

Luo J, Li R, Xu X, Niu H, Zhang Y, Wang C. SMRT and illumina RNA sequencing and characterization of a key NAC gene LoNAC29 during the flower senescence in Lilium oriental “Siberia.” Genes (Basel). 2021;12:869.

He G, Yang P, Cao Y, Tang Y, Wang L, Song M, et al. Cytokinin type-B response regulators promote bulbil initiation in Lilium lancifolium . Int J Mol Sci. 2021;22:3320.

Liu X, Wu Z, Feng J, Yuan G, He L, Zhang D, et al. A novel R2R3-MYB gene LoMYB33 from lily is specifically expressed in anthers and plays a role in pollen development. Front Plant Sci. 2021;12: 730007.

Wu Z, Li T, Liu X, Yuan G, Hou H, Teng N. A novel R2R3-MYB transcription factor LlMYB305 from Lilium longiflorum plays a positive role in thermotolerance via activating heat-protective genes. Environ Exp Bot. 2021;184: 104399.

Yuan G, Wu Z, Liu X, Li T, Teng N. Characterization and functional analysis of LoUDT1, a bHLH transcription factor related to anther development in the lily oriental hybrid Siberia ( Lilium spp.). Plant Physiol Biochem. 2021;166:1087–95.

Wang Y, Zhou Y, Wang R, Xu F, Tong S, Song C, et al. Ethylene response factor LlERF110 mediates heat stress response via regulation of LlHsfA3A expression and interaction with LlHsfA2 in lilies ( Lilium longiflorum ). Int J Mol Sci. 2022;23:16135.

Fang S, Yang C, Ali MM, Lin M, Tian S, Zhang L, et al. Transcriptome analysis reveals the molecular regularity mechanism underlying stem bulblet formation in oriental lily “Siberia”; functional characterization of the LoLOB18 gene. Int J Mol Sci. 2022;23:15246.

Yang YY, Ma B, Li YY, Han MZ, Wu J, Zhou XF, et al. Transcriptome analysis identifies key gene LiMYB305 involved in monoterpene biosynthesis in Lilium “Siberia.” Front Plant Sci. 2022;13:1021576.

Wu Z, Li T, Cao X, Zhang D, Teng N. Lily WRKY factor LlWRKY22 promotes thermotolerance through autoactivation and activation of LlDREB2B . Hortic Res. 2022;9:uhac186.

Wu Z, Li T, Zhang D, Teng N. Lily HD-Zip I transcription factor LlHB16 promotes thermotolerance by activating LlHSFA2 and LlMBF1c. Plant Cell Physiol. 2022;63:1729–44.

Li T, Wu Z, Xiang J, Zhang D, Teng N. Overexpression of a novel heat-inducible ethylene-responsive factor gene LlERF110 from Lilium longiflorum decreases thermotolerance. Plant Sci. 2022;319: 111246.

Li T, Zhou T, Liang J, Zhang D, Teng N, Wu Z. Overexpression of lily LlWRKY22 enhances multiple abiotic stress tolerances in transgenic Arabidopsis . Ornam Plant Res. 2022;2:1–10.

Wang Z, Deng J, Liang T, Su L, Zheng L, Chen H, et al. Lilium regale Wilson WRKY3 modulates an antimicrobial peptide gene, LrDef1 , during response to Fusarium oxysporum . BMC Plant Biol. 2022;22:257.

Fu Y, Li J, Wu H, Jiang S, Zhu Y, Liu C, et al. Analyses of Botrytis cinerea -responsive LrWRKY genes from Lilium regale reveal distinct roles of two LrWRKY transcription factors in mediating responses to B. cinerea . Plant Cell Rep. 2022;41:995–1012.

Zhou Y, Wang Y, Xu F, Song C, Yang X, Zhang Z, et al. Small HSPs play an important role in crosstalk between HSF-HSP and ROS pathways in heat stress response through transcriptomic analysis in lilies ( Lilium longiflorum ). BMC Plant Biol. 2022;22:202.

Chen Y, Zhao M, Wang X, Cui J, Ge W, Zhang K. Key microRNAs and target genes involved in regulating maturation in Lilium . Ornam Plant Res. 2022;2:1–14.

Westerdahl B, Riddle L, Giraud D, Kamo K. Field test of Easter lilies transformed with a rice cystatin gene for root lesion nematode resistance. Front Plant Sci. 2023;14:1134224.

Yue Y, Wang L, Li M, Liu F, Yin J, Huang L, et al. A BAHD acyltransferase contributes to the biosynthesis of both ethyl benzoate and methyl benzoate in the flowers of Lilium oriental hybrid “Siberia.” Front Plant Sci. 2023;14:1275960.

Guo Y, Guo Z, Zhong J, Liang Y, Feng Y, Zhang P, et al. Positive regulatory role of R2R3 MYBs in terpene biosynthesis in Lilium ‘Siberia.’ Hortic Plant J. 2023;9:1024–38.

Feng Y, Guo Z, Zhong J, Liang Y, Zhang P, Sun M. The LibHLH22 and LibHLH63 from Lilium ‘Siberia’ can positively regulate volatile terpenoid biosynthesis. Horticulturae. 2023;9:459.

Fang S, Lin M, Ali MM, Zheng Y, Yi X, Wang S, et al. LhANS-rr1 , LhDFR , and LhMYB114 regulate anthocyanin biosynthesis in flower buds of Lilium ‘Siberia.’ Genes (Basel). 2023;14:559.

Liu L, Xie T, Zhao Z, Zhao J, Deng M, Wen J. Involvement of a novel NAP member LbNAP in flower senescence and dehydration stress in cut lily. Sci Hortic. 2023;308: 111482.

Liu X, He L, Wu Z, Teng N. GAMYB transcription factor LoMYB65 from lily plays a vital role in pollen development. Hortic Plant J. 2023;10:223–38.

Liu Y, Wang Q, Abbas F, Zhou Y, He J, Fan Y, et al. Light regulation of LoCOP1 and its role in floral scent biosynthesis in Lilium ‘Siberia.’ Plants (Basel). 2023;12:2004.

Deng J, Wang ZE, Li WY, Chen XH, Liu DQ. WRKY11 up-regulated dirigent expression to enhance lignin/lignans accumulation in Lilium regale Wilson during response to Fusarium wilt. J Integr Agric. 2023. https://doi.org/10.1016/j.jia.2023.07.032 .

Wang X, Li Q, Zhu H, Song M, Zhang K, Ge W. Molecular mechanisms of miR172a and its target gene LbrTOE3 regulating maturation in Lilium . Planta. 2023;258:53.

Sun Y, Zang Y, Ma Y, Wang C, Song S, Sun H. Identification and functional analysis of LpNAC37 associated with somatic embryogenesis in Lilium pumilum DC. Fisch. based on transcriptome analysis. Plant Physiol Biochem. 2023;205:107964.

Wu Z, Li T, Xiang J, Teng R, Zhang D, Teng N. A lily membrane-associated NAC transcription factor LlNAC014 is involved in thermotolerance via activation of the DREB2-HSFA3 module. J Exp Bot. 2023;74:945–63.

Wu Z, Liang J, Li T, Zhang D, Teng N. A LlMYB305-LlC3H18-LlWRKY33 module regulates thermotolerance in lily. Mol Hortic. 2023;3:15.

Zhang X, Xu S, Pan X, Wu Z, Ding L, Teng N. Low LdMYB12 expression contributes to petal spot deficiency in Lilium davidii var. unicolor . Mol Genet Genomics. 2023;298:1545–57.

Bi M, Liang R, Wang J, Qu Y, Liu X, Cao Y, et al. Multifaceted roles of LhWRKY44 in promoting anthocyanin accumulation in Asiatic hybrid lilies ( Lilium spp.). Hortic Res. 2023;10:uhad167.

Feng J, Wang Y, Ge W, Zhang K, Cui J. Regulatory mechanism of the miR172e-LbrAP2 module during the vegetative growth phase transition in Lilium . Planta. 2023;259:26.

Zhang L, Wang Z, Ji S, Zhu G, Dong Y, Li J, et al. Ferric reduction oxidase in Lilium pumilum affects plant saline-alkaline tolerance by regulating ROS homeostasis. Plant Physiol Biochem. 2023;207: 108305.

Jing Y, Song Y, Ji S, Zhang L, Wang Z, Dong Y, et al. PS II subunit P in Lilium pumilum ( LpPsbP ) confers saline-alkali resistance to the plant by scavenging ROS. Int J Mol Sci. 2023;24:3311.

Wu X, Ling W, Pan Y, Yang Z, Ma J, Yang Y, et al. Functional analysis of a lily short vegetative phase ortholog in flowering transition and floral development. Plant Physiol Biochem. 2024;206: 108287.

Wu Z, Li T, Ding L, Wang C, Teng R, Xu S, et al. Lily LlHSFC2 coordinates with HSFAs to balance heat stress response and improve thermotolerance. New Phytol. 2024;241:2124–42.

Wu Z, Li T, Zhang Y, Zhang D, Teng N. HD-Zip I protein LlHOX6 antagonizes homeobox protein LlHB16 to attenuate basal thermotolerance in lily. Plant Physiol. 2024;194:1870–88.

Tong Z, Dong T, Li Q, Wang R, He J, Hong B. R2R3-type LoMYB21 affects jasmonate-regulated development and dehiscence of anthers in lily ( Lilium oriental hybrids). Hortic Plant J. 2024;10:239–51.

Liu T, Wang Y, Li X, Che H, Zhang Y. LpNAC5 positively regulates drought, salt and alkalinity tolerance of Lilium pumilum . Gene. 2024;20: 148550.

Liu X, Yan W, Liu S, Wu J, Leng P, Hu Z. LiNAC100 contributes to linalool biosynthesis by directly regulating LiLiS in Lilium ‘Siberia.’ Planta. 2024;259:73.

Cao L, Jiang F, Liu D, Zhang J, Yang T, Zhang J, et al. Genome-wide characterization of differentially expressed scent genes in the MEP control network of the flower of Lilium ‘Sorbonne.’ Mol Biotechnol. 2024. https://doi.org/10.1007/s12033-024-01063-3 .

Cao YW, Song M, Bi MM, Yang PP, He GR, Wang J, et al. Lily (Lilium spp.) LhERF4 negatively affects anthocyanin biosynthesis by suppressing LhMYBSPLATTER transcription. Plant Sci. 2024;342:112026.

Rathore DS, Mullins E. Alternative non- Agrobacterium based methods for plant transformation. In: Roberts JA, editor. Annual plant reviews online. Chichester: John Wiley & Sons; 2018. p. 891–908.

Debernardi JM, Tricoli DM, Ercoli MF, Hayta S, Ronald P, Palatnik JF, et al. A GRF-GIF chimeric protein improves the regeneration efficiency of transgenic plants. Nat Biotechnol. 2020;38:1274–9.

Zhang M, Ma X, Jin G, Han D, Xue J, Du Y, et al. A modified method for transient transformation via pollen magnetofection in Lilium germplasm . Int J Mol Sci. 2023;24:15304.

Cao X, Xie H, Song M, Lu J, Ma P, Huang B, et al. Cut-dip-budding delivery system enables genetic modifications in plants without tissue culture. Innovation (Camb). 2023;4: 100345.

PubMed   Google Scholar  

Download references

Acknowledgements

Not applicable

This work was financed by the National Natural Science Foundation of China (grant number 32302589), the Postdoctoral Science Foundation of China (2023MD744227), Shenyang Innovation Program of Seed Industry (21-110-3-12), Liaoning Province Germplasm Innovation Grain Storage Technology Special Plan (2023JH1/10200010), and the earmarked fund for CARS (CARS-23).

Author information

Authors and affiliations.

Key Laboratory of Protected Horticulture of Education Ministry, College of Horticulture, Shenyang Agricultural University, Shenyang, 110866, China

Xinyue Fan & Hongmei Sun

National and Local Joint Engineering Research Center of Northern Horticultural Facilities Design and Application Technology, Shenyang, 110866, China

Hongmei Sun

You can also search for this author in PubMed   Google Scholar

Contributions

Xinyue Fan: Writing—original draft. Hongmei Sun: Conceptualization, Writing—review & editing, Supervision, and Funding acquisition. All the authors read and approved the final version of the paper.

Corresponding author

Correspondence to Hongmei Sun .

Ethics declarations

Ethics approval and consent to participate.

This study does not address all ethical issues and has been approved by all the authors. Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

13007_2024_1246_moesm1_esm.jpg.

Additional Files 1. Fig. S1: Literature keywords related to the field of lily research that appear together in the map. Keywords that appear more than 30 times are displayed, and different colours represent different cluster.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Fan, X., Sun, H. Exploring Agrobacterium -mediated genetic transformation methods and its applications in Lilium . Plant Methods 20 , 120 (2024). https://doi.org/10.1186/s13007-024-01246-8

Download citation

Received : 17 June 2024

Accepted : 27 July 2024

Published : 09 August 2024

DOI : https://doi.org/10.1186/s13007-024-01246-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Agrobacterium tumefaciens
• Genetically modified
• Genetic transformation
• Transgenic plants
• CRISPR‒Cas9

Plant Methods

ISSN: 1746-4811

• Submission enquiries: [email protected]