File Upload Design Pattern - I

One of my recent projects added a requirement to insert and or update records from a file rather than from the UI.   Of course, all the business logic and validation that was being done for the UI had to be done on the records being updated or inserted from the file.

Good news was that  we had kept all the business and validation logic out of the controller.  So at least we were not faced with either refactoring the controller or duplicating the logic in another section of the program.

It was a good lesson for the junior programmers on the project -- as they had been chaffing at my insistence to keep business logic out of the control layer.

The file could contain thousands of records.

Mulling this over, I came up with the following design considerations.

1. Have the file be processed asynchronously. 
2. Have one transaction per record 
3. Re-use all the business logic encoded while developing the interactive screen.
4. Not have any business logic specific to the file upload that was not directly relevant to the file-upload process itself.  I.e. I did not want to introduce any business logic in the file-upload that was different than in the interactive screen.
5. Have the process that reads and writes excel files be only loosely coupled with the upload process.
6. Have the process be robust -- and stand up to server failures/restarts etc.
7. Build in some support function to cancel/suspend the process.

The process I designed was as follows:

1. All uploads will from a file downloaded by the application.
2. End User fills out a form to request a file download that he will update.
3. End User will be directed to a status page for file being processed -- it will show the file in a process of 'Preparing'.
4. The download will be processed asynchronously and on its completion and user will see a change of status on a page from preparing to 'Ready for download'.
5. End User will download the file, update it and then upload it back into the system.  The status of the file will be changed 'Validating'.
6. System will validate the file asynchronously.
7. End-user will see the results of the validation.  If the validation failed, they can download the file with the validation errors in a new column in the spreadsheet.  Correct them and re-upload.
8. Once the file clears validation, they will be allowed to submit it for processing.
9. The file changes/adds will be applied to the system.   A final status of either successful, partially successful or failed will be returned.