Let’s rewrite it
We were faced with re-writing a pre-existing large CRUD application with much legacy technology underneath. Most of the application had a few dozen forms, some with 50+ fields (online versions of paper surveys). We expected large rounds of change requests. We hoped/expected to avoid rewrite but the client was requesting dramatic change to the data model with a thick pre-existing application stack requiring a dozen artifact changes per data model change. Sounds like a perfect recipe for one nasty, tedious rewrite. This time we are going to do it right… aren’t we?
We decided to separate the project into two phases. First – prepare complete clickable interface. Second – add back-end code in phase two. Good for us, we had a spreadsheet containing a detailed description of old data model. There were two goals we had in mind when designing the solution:
- reduce amount of work needed for rewrite;
- be flexible enough to handle client feedback in phase two.
There were 4 approaches on the table:
- Code it all by hand – nah, regarding the size of repetitive work and assuming nobody wanted to take a role of code monkey, we knew we want to automate that stuff;
- Reuse old domains code and generate the code from it – we didn’t want to stick to old technology and the changes requested by client were so fundamental, that it would not bring much value;
- Dynamic UI generated at runtime based on domains – concept of dynamic scaffold sounds appealing, but we had our concerns. Growing complexity of app would require covering vast number of edge cases and/or using hard to maintain overrides. Regarding our previous experience with similar solutions in Grails and Rail’s Active Scaffold we decided not to do it that way;
- Generate the scaffold code from domain – we decided to choose this option because it combines the advantages of all above and minimizes the risk. We can reuse old model spreadsheet, we will automate the process and if generator appears too hard to maintain we can always fall back to regular coding.
How we did it
We decided to start with data and Proof of Concept before moving to generator. We briefly discussed the structure of domain documentation (based on previous version’s spreadsheet). It required defining the way to declare fields, lookups values, validations, relations, etc. The business analyst started the work on the spreadsheet with the client and the programmer prepared a small template app (domain, service, views, backing bean, and stub for security layer). When done, we validated a spreadsheet with a simple validator written in Groovy. Domain model “master” spreadsheet was shared through Dropbox for easier cooperation.
Next step was to code the generator. We generated classes for all layers of the app – views, message properties files, menus and sql scripts with lookup values. At the same time, the spreadsheet was still updated while detailing requirements. We also had to decide when to exclude a given domain from generation in case of some complex form or rule customization needed. Our goal was to keep the requirement spreadsheet consistent with the code. The easiest way to verify it was to run a generator and check whether any files were modified. Later we found that it’s also a great way of internal acceptance testing. The generator was so convenient that we were using it until final acceptance of the application by client (maintenance team presumably would not longer use the tool as maintaining it for complex business rules would be too cumbersome).
For generator we used custom Groovy scripts. The generator was seamlessly integrated with the project using great GMaven plugin. This allowed other team members to use it without installing additional software. We chose plain groovy templates for templating. They provided very powerful scripting capabilities, but the templates became a little hard to read after a while. It was tricky to find the balance between formatting of template code and indention required in output classes. Next time we may evaluate some other templating language with the better support from IDE side.
From architectural point of view we wanted to keep generated code very standard/plain and DRY. We knew that complex code can be the deal breaker for generators and WYSWIG editors (ASP.NET integration with Visual Studio is a great example of the well done job). We decided to reduce duplication in classes using inheritance and delegation. For views we used partials and abstracted repetitive code into widgets. Generated code should only contain minimal differentiating information. In the end, we are going to maintain generated code together with manually written code.
We made our generator agnostic to the project’s domain, making space for future reuse. Only the templates are specific to this particular project. In future we could also like to add support for other spreadsheet tools like Excel or Google Docs (we were using OpenOffice Calc).
We found that it’s extremely important to inform the client about the advantages and limitations of code generation solution to a large CRUD data management application. The client may be willing to sacrifice some visual fireworks in exchange for lower cost and room for change. We believe that technology never goes first, so we need to be sure that such compromises won’t affect functionality or usability. In our case this resulted in consistent interface across the whole application.
The other advantage of using a spreadsheet as the main master domain document of the project was the greatly improved feedback loop. PM/Analyst and client actively participated in creating of software. This resulted in clear and unambiguous documentation. The spreadsheet became a kind of DSL that client understands and is able to communicate with. By automating the process we also avoided simple bugs like field name typos, and we were sure that code is consistent with specification.
We were able to deliver the prototype and the final application before deadline. The client was able to introduce new forms and made significant changes in existing screens before phase two with minimal additional cost. And it was a great fun to work on it too.
Let us know about your experiences with generator and large CRUD app and if you have any questions on details.