May 5-24: Reherse testing theorie material of my university course last year.
May 10: Search FreeCAD source code for existing testing solutions that I potentially missed till now. I couldn't find anything besides lots of python unit tests.
May 12-16: Work with the Fem Workbench to get up to date about the current state of the Workbench.
May 19: Start Testing Tool Survey with the purpose of finding out which tools and frameworks are liked by the community and which aren't.
May 24: Setup development environment on Arch Linux.
May 26: Setup devlog on github pages.
May 28: Create simplified version of FreeCAD runtime components for future prototype development. Code is avaliable on GitHub. For more information please read the FreeCAD Forum post.
May 29: Publish second part of the testing tool survey concerning the C++ unit test library on FreeCADs Forum. Add more runtime components (Document and Object) to prototype repository.
May 30: Add tests to the prototype repository. Implement test runner for C++ using Google Test. Try out different configuration for test suites (multiple binaries vs single binary, combined test execution of C++ and Python, ...).
June 3: Trying to untangle the Base and App source code to figure out the best way on how to test DocumentObjects. For that I must define what exactely defines the behaviour of one Object in isolation. I'm going to share my results this week.
June 4: Write a little summery about the type system of FreeCAD and some other basics. This helps me understand the topic better and of course I'm going to share it with the FreeCAD community and if they find it helpful I can also put it on the wiki.
June 5: Do reading on cmake to figure out the build system of FreeCAD. I also digged around in the FreeCAD source a lot. After that I wrote a new CMakeLists.txt file for a (not jet written) test binary.
June 6: Write first version of a google test binary for the Fem workbench. See commit 85b290636 on my testing branch. Be aware that I rebase that branch frequently which means that you have to use --force often.
June 11: Write specification (source documentation) for the Analysis Document Object. I also deleted a few overrides that did basically nothing (implementation of the base class have the same effect). Based on that specification I wrote a few (two :P) tests and started writing a test support library that should be useful for all C++ tests in FreeCAD. See commits 1a244a88c, e7b3bc2d8, 94f60f01a.
June 11: Write specification (source documentation) for the Analysis Document Object. I also deleted a few overrides that did basically nothing (implementation of the base class have the same effect). Based on that specification I wrote a few (two :P) tests and started writing a test support library that should be useful for all C++ tests in FreeCAD. See commits 1a244a88c, e7b3bc2d8, 94f60f01a.
Unfourtionately I was sick for a week so not much happend during that time. I used the time to read up on some related stuff like testing theorie, cmake build system and I also worked a bit on my notes on the FreeCAD source. I try to make up for the lost time by working a few weekends.
June 19: Fix documentation so that it looks good in doxygen. I also wrote my own doxygen config because in my opinion the default is overly complicated. It displays so much information (that is not needed imo) that it gets hard to find stuff. It only includes documented C++ source (no undocumented stuff and no python) I currently host my version of the documentation on github. Here are two classes I documented: Fem::FemAnalysis and DocumentTest.
June 20: Implement missing FCTest functions. Add documentation and tests for FCTest library. See commits: 555c537, 8935576 and a7f6589.
June 21: Study code related to FreeCADs ViewProvider and the Part module. The Part module relies heavily on components of App and Base. This was done as a preparation for writing tests for Fem::Constraint and the Mesh framework of the Fem module.
June 24: For future tests of FemConstraint a few comperators and std::cout support was added to the FCTest library. The operators can compare opencascade vectors with FreeCADs vectors directely with ==. This is handy for google tests ASSERT_EQ and ASSERT_NE macros. The output support enables google test to print better debug messeges when a test fails (it displays the value of the two operands). See commit ae62ae9.
June 25: The DocumentTest::addObject function makes it easier and less error prone to add Document Objects for testing purpose. The function is documented and tested. See commit 0ac25dd.
June 26: Add tests for App::Constraint. App::Constraint is the base class for all constraints of the Fem module. It provides a few unitily functions and properties for its subclasses. See commit e93efd6.
June 27: Fix the build system and tests to work on ubuntu and better integrate into the existing build system. I'm developing on Arch Linux so there where a few issues. See commits 730835c and 85d6c14.
Juli 1: Assess state of python unit testing situation of the Fem Module. There are already a number of unit tests in the femtest package. State of the test is not the best because most just fail on error and don't assert the actual output.
Juli 2: Start testing progress for python command line only tests (App). Write documentation for the femtools package. See commit fd6857e.
Juli 4: Work on python documentation using a separate doxygen website (separate from the C++ API documentation) to reduce confusion when using the docs. If you visite my doc page on github you'll notice that there is now a context sensitive C++ or Python button in the menu bar which switches between the two doc instances. I also worked on a alternative using sphinx for python and doxygen for C++ which is much more promising in my opinion (use the best tools for the jobs respectively) but it isn't yet online.
Juli 7: Change organisation of C++ unit tests. The test runner and support code is now directely below src/ in the src/Test folder. I think this is a good place for now. Tests are collected via cmake object libraries which are only fully supported till cmake 3.12. I probably have to find a less elegant solution involving cached cmake variables to avoid having to bump the required cmake version from 3.3 to 3.12. See commit: a6fcff7.
Juli 9: Add python support library to FreeCAD analog to the library I created for C++. Also very small atm. See commit: 6f60a43.
Juli 10: Add python unit tests for the femtools package. They can be executed with the following command: ./bin/FreeCADCmd -t femtest_new.test_tools. See commit e371c94.
Juli 15: Add python unit tests for the femtools package. They can be executed with the following command: ./bin/FreeCADCmd -t femtest_new.test_tools. See commit e371c94.
Juli 16: Create sphinx proof of concept page for FreeCAD showcasing a mixed abbroach to documentation combining source code documentation and external documentation.
Juli 17: Begin documenting the solver framework starting with the settings module. It can be viewed on the new sphinx documentation.
Juli 18: Add structure to the sphinx documentation and document the run module of the solver famework.
Juli 22: Next step is to create unit tests that execute a simulation and inspect the results. Using VTK to inspect the results would be supported by all solvers at the moment. Consider other ways and make a plan for implementing result checking.
Juli 23: Read VTK API documentation to find out how to best extract relevant information from VTK results. Also search FreeCAD source for existing solutions. DataAtPointFilter looks promising.
Juli 24: Debug DataAtPointFilter segmentation fault problems. Seems to have something to do with out of sync VTK pipeline execution and FreeCADs recompute system.
Juli 25: Write test support library for the Fem module. It contains a unitest derived class whose purpose is to facilitate unittests that execute a simulation and assert the result data. See commit c7dd20a.
Juli 29: Debug DataAtPointFilter update issue when using the test support library. Fixed it by using recompute at the right position. The VTK filter seems to be very sensitive regarding recomputes.
Juli 30: Research analytic solution for simple problems that could be used for the tests. The analytic results are used to determine if the solver calculated the right result. The problem I choose is the cantilever end load problem which is similar to the FEM example FCStd.
Juli 31: Write Cantilever End Load unittest for elmer using the support library. See commit c5ed907.
August 1: Test the same Cantilever End Load test with Calculix. Also added a second simulation type: A Ccntilever with a uniform load on the top face. This is also tested with both elmer and Calculix. See commit c5ed907.
August 2: Make minor adjustments to the Fem source to make testing easier and less confusing. Replaced or removed a bunch of output on stdout and adjust a few method signatures. See commits 8c79422, 9f2e3b5, 15e589d and d1722fc.
August 5: Evaluate QTest for FreeCAD Gui Testing. Make TaskPanel unitest prototype using unittest module and QTest.
August 6: Add python gui testing library (4d767c3) only containing stuff to test task panels atm. Also add a showcase test case testing the task panel of gmsh (8aa8075).
August 7: Improve gui test support library to better fit the needs of the test cases. Also adapted the gmsh gui tests to work with the new support library. See commit: (3910af9)
August 8: Integrate new tests into the existing structure and than develop a new structure that provides a similar workflow for C++ and Python tests. The commits are not yet online because I have to compile a lot of versions which takes a few hours. Also had to spend some time fixing simulation tests.
August 12: Develop strategy for in-place C++ test execution (executing tests out of a running instance of FreeCAD. Create a proof of concept separate from FreeCAD.
August 13: Debug various oddities and bugs of the proof of concept which apear to be related to the fact that google test is not intended to be used for in-place test execution.
August 14: Contiunue the work on the proof of concept for google test. Unfourtionately I discovered that it is not possible to initialice google test twice in one program execution which essentially makes it impossible to execute tests out of a running instance (it would only be possible to do the same test run again and again). A different unit test library must be used instead of google test.
August 15: Start work on python test coverage. Read documentation of coverage.py and play around with a few examples.
August 19: Integrate coverage.py into the FreeCADTest.py script which is used when executing tests from the command line (-t option). A report summery is printed at the end of the test output and a html report is generated in the ./coverage/ directory when the new --cov option is used.
August 20: Restructure python tests of the Fem workbench. Packages can now be specified directly to execute a collection of tests and also contain information defining which source files should be covered by the tests (for useful coverage reports).
August 21: Add new option --log to define the log level on the command line. This is useful to control the loglevel on test runs. Also replace all print calls with Console.Print* calls in the Fem module.
August 22: Tidy up commit history and prepare branches for the final report. Create final report page on gitpages and submit the final submission.