Overview
After being selected by a major Consumer Packaged Goods (CPG) manufacturer, we were tasked with providing all the automation and control system design and development to create a large greenfield (new) facility. The key components of the initiative included leveraging our standardized project execution processes to deliver an overall detailed design, leveraging an agile development methodology for the total control, Supervisory Control and Data Acquisition (SCADA), and Manufacturing Execution System (MES) integrated solution. After completion, formal testing followed with an IO checkout, dry checkout, commissioning, and then ended by supporting our customer and training their employees. The whole process took 2.5 years to complete.
The Avanceon Solution
In a complex and long-term project such as this one, there are many components and phases of the development. The following section highlights the major phases along with the focus and value provided in each step.
Detailed Design: The intent was to deliver the functional design specifications for each area of the project. This included unloading, processing, blending, and packaging of the product. Formal documents were developed and presented to the stakeholders for further comprehension of how the system would function. Upon completion, the documents were used by the development team for implementation.
*Note: A key component of the Detailed Design phase is to deliver documents that include simple, text-based instructions that capture the steps, actions, and exceptions required to develop the system. Below is a sample of one of the produced documents:
Each document is laid out in a table format to further relay the information. From this table, it was clear what devices were involved in the operation and how they needed to be programmed/configured. Additionally, it was discovered what the phases and control modules went along with what actions were needed at each step. By the end of the design phase there was over 500 pages of documentation in directly useable requirements and design information that was ready for development and programming.
Agile Development: Adding to the detailed design, the requirements dictated the addition of four individual Programmable Logic Controllers (ControlLogix 5580 P) to control and manage the unloading, processing, blending, and packaging functions for the plant. The objective was to deliver FactoryTalk (FT) View studio distributed Human Machine Interface (HMI) applications and FT Batch solution flexible enough to meet the needs of the batching demands for of the facility. The purpose was to lay down a foundation for the PLC, HMI, and batch standards with the intent to use it for further expansions at this site and future sites for the customer. It should be noted that the focus was on developing modular programming standards concentrated on the device, phase, and control modules (ultimately, this simplified revisions and future enhancements). The designed batch and control structure allowed easier additions of new formulas and other processes and products.
The Agile Process was accomplished by sprints of small pilot developments i.e., specific use cases to demonstrate the standards and approach used during this project. By focusing the standards on the process areas and phase/control modules, the resultant deliverable complemented the inherit functionality of the FT Batch S88 batch software and seamlessly married it to a best practice implementation of the HMI and PLC layers (PlantPAX). The project had regularly scheduled Sprint Reviews for collaboration and approval by using simulated PLC logic and the actual HMI application. From a project management perspective, the project included regular status updates with all project stakeholders and a broader operational audience. The goal was to ensure the information was shared and would continue to be shared with issues and enhancements resolved as they came to light.
Through the Agile Development process a few highlights surfaced that were big wins for the customer and the end user. The PlantPAX 5.0 implementation and ControlLogix 5580 P Processors yielded new and enhanced PlantPAX device standards and PhaseManager controls with both including PLC logic and HMI faceplates. The PlantPAX HMI layouts utilize a high-performance HMI approach with VBA and database customized elements inside FT View that allowed for an automated batch queueing function as well as packaging unit Op management, and complex process area setpoint management system. Specific advantages of the PlantPax objects, PlantPax HML layouts, and FT Batch are as follows:
PlantPAX Objects
• Standards defined to reduce development time
• Integrated runtime, stroke count, and other performance functions into the device standard
• Permissive and Interlocks status integrated into device templates
• Integrated multi-tiered security levels
• Standardized PLC logic across application
PlantPAX HMI Layouts
• Sleek and consistent look and feel
• Clean navigation with standard approach
• High Performance/Situational Awareness approach that uses color to draw attention to important and pertinent information/issues
• Hierarchical Screen Layout
Level 1 – Area Overview
Level 2 – Operation Screens
Level 3 – Unit Screens
Level 4 – Device and Phase Control
FT Batch
• Tight integration between FT Batch and PLC Phase Manager
• Modular Phase approach
One function per phase for simplicity and troubleshooting
Only include devices relevant to the function
• No PLC changes required to change sequence of operations in the recipe
• Historical reporting of each batch
• Key Enhancement – Automatic Batch Queuing based on required number of pallets of finish goods
User Experience Features – from a user experience perspective the project included multiple dual monitor HMI stations positioned throughout the plant with mobile HMI access via iPads and IT approved mobile devices.
Interfacing the MES: The system has a MES system for key plant functions such as OEE, quality, product traceability, and overall workflow. The MES interfaces with the control system, SCADA, and Process Historian to expose key data points, statuses, and calculations for consumption by the various functions. The objective was to consolidate the data in the control layer for the MES, allowing it to align with the corporate standard MES modules. To collaborate with the customer’s MES vendor, Avanceon assigned a specific Subject Matter Expert (SME) for the data interface scope to be a single point of contact to identify the required data points and present them to the MES application. In order to manage and keep track of the volume of data needed by the application an MES Functional Design Specification (FDS) was developed to describe each data point along with any additional parameters. For the purpose of accomplishing the data interface, new routines were created within the PLC to handle the exchange. These routines were separate from the core process control programs to allow for ease of programming and troubleshooting. A key feature of the integration of the FT Batch ingredient consumption to the MES. Through this interface, MES triggered the completion of a work order and listed the product for the next work order. By following this work order through the process, the system sent the required packaging unit operation configuration and setpoints to the downstream equipment. Once set up, configured, and production was running, key performance data was looped back to the MES to track OEE and Quality.
Formal Testing: In order to gain operational buy in and manage the impact of new people and a new system, the project had an extensive testing and demonstration phase. Once the deliverables were field ready during this phase, they presented a final opportunity for stakeholders to make corrections and add enhancements to the overall process. To accomplish this, the four PLC programs: unloading, processing, blending, and packaging had to work correctly. In this testing process the FT View HMI application was fully configured, FT Batch instance operational, and the FAT protocol by area were executed in a simulated environment. By the end, it was confirmed that everything was in working order.
To achieve this, integrated testing of all components occurred as follows:
1. A simulated PreFAT was implemented to allow things to be exercised and tested as a dress rehearsal before performing the test in front of the customer.
2. FAT: A series of formal tests with customer stakeholders was held. Since Sprint Reviews were used during the Agile Development the stakeholders were acquainted with the system and, at this point, the system had been thoroughly exercised. The FAT served as a final review and acceptance of the system.
Field Checkout: Field checkout and commissioning is broken down into two subcategories: IO Checkout and Dry Checkout.
IO Checkout: Dry Checkout ensures the IO was installed correctly and operates as expected. To accomplish this each IO point or device is physically and electrically verified. Progress was tracked via area checkout sheets and operation was confirmed from the device, through the PLC to the HMI. Any deviations are tracked via a detailed punchlist with a record of the device status, issues, and resolutions.
Dry Checkout: In an effort to confirm the system is ready for wet commissioning a dry check is performed. This step expands on the progress from IO check and verifies the system is in conformance functionally as defined in the FDS. Similar to the IO Checkout, formal checkout sheets are leveraged to track progress and the system is verified electrically, mechanically, and through to the HMI. Any deviations are tracked the punchlist with a record of the function status, issues, and resolutions.
Training: In parallel to checkout, training is provided to plant teams to be able to operate and maintain the new system. This includes operator, maintenance, and engineering level training with an emphasis on troubleshooting. To provide reinforcement and post training access, the training material is converted to online training courses that the customer can review for on the spot refreshers. All courses are developed as small dose, focused job aids for each area of operation. The material from the course is available in a central repository for consumption as needed in the future.
Commissioning: The culmination of the design, development, and testing is the commissioning phase. In this final project step the overall system is tested for production conditions with water or some other non-critical, inexpensive components. The system is exercised against the FDS and is run as it would in during a typical workday. In this phase on the job training for plant operations and maintenance occurs and compliments formal classroom training.
Post Commissioning 24×7 Support: In order to augment the plant resources and help them manage the new system operation Avanceon provides 24×7 support to address any question or emergency that might arise. The support services team provides immediate solutions to the majority of issues encountered. In the event of a serious or unknown issue, the support team has direct access to the project SMEs and team members. With regards to handling any enhancements or on the spot training needs, blocks of hours are included in this phase. Inside of support the project development environment is maintained for future investigation and enhancement testing purposes. Post-production support provides collaboration with the plant to ensure adoption of the system and that it is used to its fullest potential.
Result
At the end of the project the customer’s plant and corporate team were very impressed and pleased to realize the vision and goal that was set over two years prior. With the new plant’s production capacity the customer has the ability to supply a larger geography at a lower cost for production and shipping. The plant intends to add more packaging capacity to expand the output and with the structure set in place during the original project, the control and SCADA system is ready to accommodate the addition. Furthermore, the PlantPAX interface is intuitive for operators to use and provides the information they need to make decisions in real time and the flexible batching system allows control of formulas and clear management of each production batch. The plant represents the most technically advanced system in the customer’s network of facilities and is a model for future implementations.