Developing, testing, building and maintaining an Eclipse project based on EMF and Xtext

The study addresses a recurring practical problem: how to obtain high confidence in the correctness of an Eclipse/Xtext-based DSL toolchain—spanning language services, IDE integration, and UI workflows—while keeping builds and releases reproducible and CI-friendly across platforms. We present a general, practice-oriented testing and CI framework for such toolchains and instantiate it in Edelta, an open-source framework and Xtext-based DSL for the evolution and refactoring of EMF metamodels and models. We describe a comprehensive testing strategy, based on test-driven development, spanning fast unit tests for the runtime and refactoring libraries, headless DSL tests for parsing, scoping, validation, type checking, interpretation, and code generation, IDE integration tests for the Eclipse editor, and end-to-end SWTBot tests that exercise complete user workflows. These tests are orchestrated in a single Maven/Tycho build and continuously executed on GitHub Actions across platforms, with SonarCloud providing code-quality and coverage feedback. Our CI setup also builds and verifies distributable Eclipse products and a self-contained p2 update site. An empirical analysis of a representative CI run shows a clear testing-pyramid distribution, with most checks implemented at the fastest layers, keeping the overall CI turnaround time low while providing high confidence in correctness. We summarize lessons learned, actionable guidelines, and pitfalls to avoid when engineering Eclipse- and Xtext-based research software that must be both feature-rich and continuously releasable.