Updated documentation + backend to enable the dashboard

README.md

@@ -15,6 +15,7 @@ Unitgrade is an automatic report and exam evaluation framework that enables inst
     - Instructors can automatically verify the students solution using a Docker VM and run hidden tests
  - Automatic Moss anti-plagiarism detection
  - CMU Autolab integration (Experimental)
+ - A live dashboard which shows the outcome of the tests
 
 ### Install
 Simply use `pip`
@@ -30,6 +31,7 @@ The figure shows an overview of the workflow.
  - You write exercises and a suite of unittests. 
  - They are then compiled to a version of the exercises without solutions. 
  - The students solve the exercises using the tests and when they are happy, they run an automatically generated `_grade.py`-script to produce a `.token`-file with the number of points they obtain. This file is then uploaded for further verification/evaluation.
+ - The students can see their progress and review hints using the dashboard (see below)
 
 ### Videos
 Videos where I try to talk and code my way through the examples can be found on youtube:
@@ -64,7 +66,7 @@ instructor/cs101/deploy.py   # A private file to deploy the tests
 ### The homework
 The homework is just any old python code you would give to the students. For instance:
 ```python
-# example_simplest/instructor/cs101/homework1.py
+# autolab_example_py_upload/instructor/cs102_autolab/homework1.py
 def reverse_list(mylist): #!f 
     """
     Given a list 'mylist' returns a list consisting of the same elements in reverse order. E.g.
@@ -75,10 +77,9 @@ def reverse_list(mylist): #!f
 def add(a,b): #!f
     """ Given two numbers `a` and `b` this function should simply return their sum:
     > add(a,b) = a+b """
-    return a+b
+    return a+b*2
 
-if __name__ == "__main__":
-    # Example usage:
+if __name__ == "__main__": # Example usage:
     print(f"Your result of 2 + 2 = {add(2,2)}")
     print(f"Reversing a small list", reverse_list([2,3,5,7])) 
 ```
@@ -119,7 +120,12 @@ class Report1(Report):
     pack_imports = [cs101]     # Include all .py files in this folder
 
 if __name__ == "__main__":
-    evaluate_report_student(Report1()) 
+    # from HtmlTestRunner import HTMLTestRunner
+    import HtmlTestRunner
+    unittest.main(testRunner=HtmlTestRunner.HTMLTestRunner(output='example_dir'))
+
+
+    # evaluate_report_student(Report1()) 
 ```
 
 ### Deployment
@@ -168,6 +174,32 @@ This runs an identical set of tests and produces the file `Report1_handin_10_of_
  - You can easily use the framework to include output of functions. 
  - See below for how to validate the students results 
 
+
+### Viewing the results using the dashboard
+I recommend to monitor and run the tests from the IDE, as this allows you to use the debugger in conjunction with your tests. 
+However, unitgrade comes with a dashboard that allows students to see the outcome of individual tests 
+ and what is currently recorded in the `token`-file. To start the dashboard, they should simply run the command
+```
+unitgrade
+```
+from a directory that contains a test (the directory will be searched recursively for test files). 
+ The command will start a small background service and open a webpage:
+
+![The dashboard](https://gitlab.compute.dtu.dk/tuhe/unitgrade/-/raw/master/docs/dashboard.png)
+
+Features supported in the current version:
+ - Shows which files need to be edited to solve the problem
+ - Collect hints given in the homework files and display them for the relevant tests
+ - fully responsive -- the UI, including the terminal, will update while the test is running regardless of where you launch the test
+ - Allows students to re-run tests from the UI
+ - Shows current test status and results captured in `.token`-file
+ - Tested on Windows/Linux 
+ - Frontend is pure javascript and the backend only depends on python packages. 
+
+The frontend is automatically enabled the moment your classes inherits from the `UTestCase`-class; no configuration files required, and there are no known bugs. 
+
+Note the frontend is currently not provided in the pypi `unitgrade` package, but only through the gitlab repository (install using `git clone` and then `pip install -e ./`) -- it seems ready, but I want to test it on mac and a few more systems before publishing it. 
+
 ## How safe is Unitgrade?
 There are three principal ways of cheating:
  - Break the framework and submit a `.token` file that 'lies' about the true number of points
@@ -197,13 +229,19 @@ One of the main advantages of `unitgrade` over web-based autograders it that tes
 # example_framework/instructor/cs102/report2.py
 from unitgrade import UTestCase, cache  
 
+
+
 class Week1(UTestCase):
+    @classmethod
+    def setUpClass(cls) -> None:
+        a = 234
+
     def test_add(self):
         self.assertEqualC(add(2,2))
         self.assertEqualC(add(-100, 5))
 
-    def test_reverse(self):
-        self.assertEqualC(reverse_list([1, 2, 3])) 
+    # def test_reverse(self):
+    #     self.assertEqualC(reverse_list([1, 2, 3])) 
 ```
 Note we have changed the test-function to `self.assertEqualC` (the `C` is for cache) and dropped the expected result. What `unitgrade` will do
 is to evaluate the test *on the working version of the code*, compute the results of the test, 
@@ -213,21 +251,21 @@ is to evaluate the test *on the working version of the code*, compute the result
 Titles can be set either using python docstrings or programmatically. An example:
 ```python 
 # example_framework/instructor/cs102/report2.py
-class Week1Titles(UTestCase): 
-    """ The same problem as before with nicer titles """
-    def test_add(self):
-        """ Test the addition method add(a,b) """
-        self.assertEqualC(add(2,2))
-        print("output generated by test")
-        self.assertEqualC(add(-100, 5))
-        # self.assertEqual(2,3, msg="This test automatically fails.")
-
-    def test_reverse(self):
-        ls = [1, 2, 3]
-        reverse = reverse_list(ls)
-        self.assertEqualC(reverse)
-        # Although the title is set after the test potentially fails, it will *always* show correctly for the student.
-        self.title = f"Checking if reverse_list({ls}) = {reverse}"  # Programmatically set the title 
+# class Week1Titles(UTestCase): 
+#     """ The same problem as before with nicer titles """
+#     def test_add(self):
+#         """ Test the addition method add(a,b) """
+#         self.assertEqualC(add(2,2))
+#         print("output generated by test")
+#         self.assertEqualC(add(-100, 5))
+#         # self.assertEqual(2,3, msg="This test automatically fails.")
+#
+#     def test_reverse(self):
+#         ls = [1, 2, 3]
+#         reverse = reverse_list(ls)
+#         self.assertEqualC(reverse)
+#         # Although the title is set after the test potentially fails, it will *always* show correctly for the student.
+#         self.title = f"Checking if reverse_list({ls}) = {reverse}"  # Programmatically set the title 
 ```
 When this is run, the titles are shown as follows:
 ```terminal
@@ -236,7 +274,7 @@ When this is run, the titles are shown as follows:
 | | | |_ __  _| |_| |  \/_ __ __ _  __| | ___ 
 | | | | '_ \| | __| | __| '__/ _` |/ _` |/ _ \
 | |_| | | | | | |_| |_\ \ | | (_| | (_| |  __/
- \___/|_| |_|_|\__|\____/_|  \__,_|\__,_|\___| v0.1.17, started: 19/05/2022 15:14:09
+ \___/|_| |_|_|\__|\____/_|  \__,_|\__,_|\___| v0.1.22, started: 15/06/2022 09:18:15
 
 CS 102 Report 2 
 Question 1: Week1                                                                                                       
@@ -250,9 +288,10 @@ Question 2: The same problem as before with nicer titles
  * q2.2) Checking if reverse_list([1, 2, 3]) = [3, 2, 1]............................................................PASS
  * q2)   Total...................................................................................................... 6/6
  
-Total points at 15:14:09 (0 minutes, 0 seconds)....................................................................16/16
+Total points at 09:18:16 (0 minutes, 0 seconds)....................................................................16/16
 
 Including files in upload...
+path.: _NamespacePath(['C:\\Users\\tuhe\\Documents\\unitgrade_private\\examples\\example_framework\\instructor\\cs102', 'C:\\Users\\tuhe\\Documents\\unitgrade_private\\examples\\example_framework\\instructor\\cs102'])
  * cs102
 > Testing token file integrity...
 Done!
@@ -267,21 +306,21 @@ What happens behind the scenes when we set `self.title` is that the result is pr
 The `@cache`-decorator offers a direct ways to compute the correct result on an instructors computer and submit it to the student. For instance:
 ```python
 # example_framework/instructor/cs102/report2.py
-class Question2(UTestCase): 
-    @cache
-    def my_reversal(self, ls):
-        # The '@cache' decorator ensures the function is not run on the *students* computer
-        # Instead the code is run on the teachers computer and the result is passed on with the
-        # other pre-computed results -- i.e. this function will run regardless of how the student happens to have
-        # implemented reverse_list.
-        return reverse_list(ls)
-
-    def test_reverse_tricky(self):
-        ls = (2,4,8)
-        ls2 = self.my_reversal(tuple(ls))                   # This will always produce the right result, [8, 4, 2]
-        print("The correct answer is supposed to be", ls2)  # Show students the correct answer
-        self.assertEqualC(reverse_list(ls))                 # This will actually test the students code.
-        return "Buy world!"                                 # This value will be stored in the .token file  
+# class Question2(UTestCase): 
+#     @cache
+#     def my_reversal(self, ls):
+#         # The '@cache' decorator ensures the function is not run on the *students* computer
+#         # Instead the code is run on the teachers computer and the result is passed on with the
+#         # other pre-computed results -- i.e. this function will run regardless of how the student happens to have
+#         # implemented reverse_list.
+#         return reverse_list(ls)
+#
+#     def test_reverse_tricky(self):
+#         ls = (2,4,8)
+#         ls2 = self.my_reversal(tuple(ls))                   # This will always produce the right result, [8, 4, 2]
+#         print("The correct answer is supposed to be", ls2)  # Show students the correct answer
+#         self.assertEqualC(reverse_list(ls))                 # This will actually test the students code.
+#         return "Buy world!"                                 # This value will be stored in the .token file  
 ```
 The `@cache` decorator will make sure the output of the function is pre-computed when the test is set up, and that the function will 
 simply return the correct result regardless of the function body. This is very helpful in a few situations:
@@ -503,26 +542,30 @@ The code for the example can be found in `examples/autolab_example`. It consists
 
 Concretely, the following code will download and build the image (note this code must be run on the same machine that you have installed Autolab on)
 ```python
-# autolab_token_upload/deploy_autolab.py
+# autolab_example_py_upload/instructor/cs102_autolab/deploy_autolab.py
     # Step 1: Download and compile docker grading image. You only need to do this once.  
-    download_docker_images("./docker") # Download docker images from gitlab (only do this once.
-    dockerfile = f"./docker/docker_tango_python/Dockerfile"
-    autograde_image = 'tango_python_tue'
-    compile_docker_image(Dockerfile=dockerfile, tag=autograde_image)  # Compile docker image. 
+    download_docker_images("../docker") # Download docker images from gitlab (only do this once).
+    dockerfile = f"../docker/docker_tango_python/Dockerfile"
+    autograde_image = 'tango_python_tue2'  # Tag given to the image in case you have multiple images.
+    compile_docker_image(Dockerfile=dockerfile, tag=autograde_image, no_cache=False)  # Compile docker image. 
 ```
 Next, simply call the framework to compile any `_grade.py`-file into an Autolab-compatible `.tar` file that can be imported from the web interface. The script requires you to specify 
 both the instructor-directory and the directory with the files the student have been handed out (i.e., the same file-system format we have seen earlier). 
 ```python
-# autolab_token_upload/deploy_autolab.py
+# autolab_example_py_upload/instructor/cs102_autolab/deploy_autolab.py
     # Step 2: Create the cs102.tar file from the grade scripts. 
-    instructor_base = f"../example_framework/instructor"
-    student_base = f"../example_framework/students"
-    output_tar = deploy_assignment("cs102",  # Autolab name of assignment (and name of .tar file)
+    instructor_base = f"."
+    student_base = f"../../students/cs102_autolab"
+
+    from report2_test import Report2
+    # INSTRUCTOR_GRADE_FILE =
+    output_tar = new_deploy_assignment("cs105h",  # Autolab name of assignment (and name of .tar file)
                                    INSTRUCTOR_BASE=instructor_base,
-                                   INSTRUCTOR_GRADE_FILE=f"{instructor_base}/cs102/report2_grade.py",
+                                   INSTRUCTOR_GRADE_FILE=f"{instructor_base}/report2_test_grade.py",
                                    STUDENT_BASE=student_base,
-                                   STUDENT_GRADE_FILE=f"{student_base}/cs102/report2_grade.py",
-                                   autograde_image_tag=autograde_image) 
+                                   STUDENT_GRADE_FILE=f"{instructor_base}/report2_test.py",
+                                   autograde_image_tag=autograde_image,
+                                   homework_file="homework1.py") 
 ```
 This will produce a file `cs102.tar`. Whereas you needed to build the Docker image on the machine where you are running Autolab, you can build the lab assignments on any computer.
 ### Step 3: Upload the `.tar` lab-assignment file 
@@ -548,9 +591,9 @@ and TAs can choose to annotate the students code directly in Autolab -- we are h
 # Citing
 ```bibtex
 @online{unitgrade_devel,
-	title={Unitgrade-devel (0.1.39): \texttt{pip install unitgrade-devel}},
+	title={Unitgrade-devel (0.1.42): \texttt{pip install unitgrade-devel}},
 	url={https://lab.compute.dtu.dk/tuhe/unitgrade_private},
-	urldate = {2022-06-15}, 
+	urldate = {2022-09-16}, 
 	month={9},
 	publisher={Technical University of Denmark (DTU)},
 	author={Tue Herlau},

devel/example_devel/instructor/cs108/unitgrade_data/AnotherTest-test_even_more.json

-{"state": "pass", "run_id": 863304, "coverage_files_changed": null, "stdout": [[0, "Dashboard> Evaluation completed."]]}
\ No newline at end of file

devel/example_devel/instructor/cs108/unitgrade_data/AnotherTest-test_more.json

-{"state": "pass", "run_id": 282722, "coverage_files_changed": null, "stdout": [[0, "Dashboard> Evaluation completed."]]}
\ No newline at end of file

devel/example_devel/instructor/cs108/unitgrade_data/Numpy-setUpClass.json

-{"run_id": 188727, "coverage_files_changed": null, "stdout": [[0, "Dum di dai, I am running some setup code here.\nHello world 0\nHello world 1\nHello world 2\nHello world 3\nHello world 4\nHello world 5\nHello world 6\nHello world 7\nHello world 8\nHello world 9\nSet up.\nDashboard> Evaluation completed."]], "state": "pass"}
\ No newline at end of file

devel/example_devel/instructor/cs108/unitgrade_data/Numpy-test_bad.json

-{"state": "fail", "run_id": 1789, "coverage_files_changed": null, "stdout": [[0, "\u001b[31m\r  0%|          | 0/100 [00:00<?, ?it/s]\u001b[37m"], [1, "\u001b[31m\r 10%|#         | 10/100 [00:00<00:00, 97.25it/s]\u001b[37m\u001b[31m\r 20%|##        | 20/100 [00:00<00:00, 97.27it/s]\u001b[37m"], [2, "\u001b[31m\r 30%|###       | 30/100 [00:00<00:00, 95.97it/s]\u001b[37m"], [3, "\u001b[31m\r 40%|####      | 40/100 [00:00<00:00, 95.72it/s]\u001b[37m\u001b[31m\r 50%|#####     | 50/100 [00:00<00:00, 93.34it/s]\u001b[37m"], [4, "\u001b[31m\r 60%|######    | 60/100 [00:00<00:00, 91.76it/s]\u001b[37m\u001b[31m\r 70%|#######   | 70/100 [00:00<00:00, 93.45it/s]\u001b[37m"], [5, "\u001b[31m\r 80%|########  | 80/100 [00:00<00:00, 94.95it/s]\u001b[37m\u001b[31m\r 90%|######### | 90/100 [00:00<00:00, 95.62it/s]\u001b[37m"], [6, "\u001b[31m\r100%|##########| 100/100 [00:01<00:00, 95.82it/s]\u001b[37m\u001b[31m\u001b[37m\u001b[31m\r100%|##########| 100/100 [00:01<00:00, 94.89it/s]\u001b[37m\u001b[31m\n\u001b[37m\u001b[92m>\n\u001b[92m> Hints (from 'test_bad')\n\u001b[92m>   * Remember to properly de-indent your code.\n>   * Do more stuff which works.\n\u001b[31mTraceback (most recent call last):\n  File \"/usr/lib/python3.10/unittest/case.py\", line 59, in testPartExecutor\n    yield\n  File \"/usr/lib/python3.10/unittest/case.py\", line 591, in run\n    self._callTestMethod(testMethod)\n  File \"/home/tuhe/Documents/unitgrade/src/unitgrade/framework.py\", line 534, in _callTestMethod\n    res = testMethod()\n  File \"/home/tuhe/Documents/unitgrade_private/devel/example_devel/instructor/cs108/report_devel.py\", line 67, in test_bad\n    self.assertEqual(1, d['x1'])\nAssertionError: 1 != 100\n\u001b[37mDashboard> Evaluation completed."]], "wz_stacktrace": "<div class=\"traceback\">\n  <h3>Traceback <em>(most recent call last)</em>:</h3>\n  <ul><li><div class=\"frame\" id=\"frame-140582372419264\">\n  <h4>File <cite class=\"filename\">\"/usr/lib/python3.10/unittest/case.py\"</cite>,\n      line <em class=\"line\">59</em>,\n      in <code class=\"function\">testPartExecutor</code></h4>\n  <div class=\"source library\"><pre class=\"line before\"><span class=\"ws\">    </span>@contextlib.contextmanager</pre>\n<pre class=\"line before\"><span class=\"ws\">    </span>def testPartExecutor(self, test_case, isTest=False):</pre>\n<pre class=\"line before\"><span class=\"ws\">        </span>old_success = self.success</pre>\n<pre class=\"line before\"><span class=\"ws\">        </span>self.success = True</pre>\n<pre class=\"line before\"><span class=\"ws\">        </span>try:</pre>\n<pre class=\"line current\"><span class=\"ws\">            </span>yield</pre>\n<pre class=\"line after\"><span class=\"ws\">        </span>except KeyboardInterrupt:</pre>\n<pre class=\"line after\"><span class=\"ws\">            </span>raise</pre>\n<pre class=\"line after\"><span class=\"ws\">        </span>except SkipTest as e:</pre>\n<pre class=\"line after\"><span class=\"ws\">            </span>self.success = False</pre>\n<pre class=\"line after\"><span class=\"ws\">            </span>self.skipped.append((test_case, str(e)))</pre></div>\n</div>\n\n<li><div class=\"frame\" id=\"frame-140582372597696\">\n  <h4>File <cite class=\"filename\">\"/usr/lib/python3.10/unittest/case.py\"</cite>,\n      line <em class=\"line\">591</em>,\n      in <code class=\"function\">run</code></h4>\n  <div class=\"source library\"><pre class=\"line before\"><span class=\"ws\">                </span>with outcome.testPartExecutor(self):</pre>\n<pre class=\"line before\"><span class=\"ws\">                    </span>self._callSetUp()</pre>\n<pre class=\"line before\"><span class=\"ws\">                </span>if outcome.success:</pre>\n<pre class=\"line before\"><span class=\"ws\">                    </span>outcome.expecting_failure = expecting_failure</pre>\n<pre class=\"line before\"><span class=\"ws\">                    </span>with outcome.testPartExecutor(self, isTest=True):</pre>\n<pre class=\"line current\"><span class=\"ws\">                        </span>self._callTestMethod(testMethod)</pre>\n<pre class=\"line after\"><span class=\"ws\">                    </span>outcome.expecting_failure = False</pre>\n<pre class=\"line after\"><span class=\"ws\">                    </span>with outcome.testPartExecutor(self):</pre>\n<pre class=\"line after\"><span class=\"ws\">                        </span>self._callTearDown()</pre>\n<pre class=\"line after\"><span class=\"ws\"></span> </pre>\n<pre class=\"line after\"><span class=\"ws\">                </span>self.doCleanups()</pre></div>\n</div>\n\n<li><div class=\"frame\" id=\"frame-140582372597808\">\n  <h4>File <cite class=\"filename\">\"/home/tuhe/Documents/unitgrade/src/unitgrade/framework.py\"</cite>,\n      line <em class=\"line\">534</em>,\n      in <code class=\"function\">_callTestMethod</code></h4>\n  <div class=\"source \"><pre class=\"line before\"><span class=\"ws\">        </span>self._ensure_cache_exists()  # Make sure cache is there.</pre>\n<pre class=\"line before\"><span class=\"ws\">        </span>if self._testMethodDoc is not None:</pre>\n<pre class=\"line before\"><span class=\"ws\">            </span>self._cache_put((self.cache_id(), &#39;title&#39;), self.shortDescriptionStandard())</pre>\n<pre class=\"line before\"><span class=\"ws\"></span> </pre>\n<pre class=\"line before\"><span class=\"ws\">        </span>self._cache2[(self.cache_id(), &#39;assert&#39;)] = {}</pre>\n<pre class=\"line current\"><span class=\"ws\">        </span>res = testMethod()</pre>\n<pre class=\"line after\"><span class=\"ws\">        </span>elapsed = time.time() - t</pre>\n<pre class=\"line after\"><span class=\"ws\">        </span>self._get_outcome()[ (self.cache_id(), &#34;return&#34;) ] = res</pre>\n<pre class=\"line after\"><span class=\"ws\">        </span>self._cache_put((self.cache_id(), &#34;time&#34;), elapsed)</pre>\n<pre class=\"line after\"><span class=\"ws\"></span> </pre>\n<pre class=\"line after\"><span class=\"ws\"></span> </pre></div>\n</div>\n\n<li><div class=\"frame\" id=\"frame-140582372597920\">\n  <h4>File <cite class=\"filename\">\"/home/tuhe/Documents/unitgrade_private/devel/example_devel/instructor/cs108/report_devel.py\"</cite>,\n      line <em class=\"line\">67</em>,\n      in <code class=\"function\">test_bad</code></h4>\n  <div class=\"source \"><pre class=\"line before\"><span class=\"ws\">        </span># for i in range(10):</pre>\n<pre class=\"line before\"><span class=\"ws\">        </span>from tqdm import tqdm</pre>\n<pre class=\"line before\"><span class=\"ws\">        </span>for i in tqdm(range(100)):</pre>\n<pre class=\"line before\"><span class=\"ws\">            </span># print(&#34;The current number is&#34;, i)</pre>\n<pre class=\"line before\"><span class=\"ws\">            </span>time.sleep(.01)</pre>\n<pre class=\"line current\"><span class=\"ws\">        </span>self.assertEqual(1, d[&#39;x1&#39;])</pre>\n<pre class=\"line after\"><span class=\"ws\">        </span>for b in range(10):</pre>\n<pre class=\"line after\"><span class=\"ws\">            </span>self.assertEqualC(add(3, b))</pre>\n<pre class=\"line after\"><span class=\"ws\"></span> </pre>\n<pre class=\"line after\"><span class=\"ws\"></span> </pre>\n<pre class=\"line after\"><span class=\"ws\">    </span>def test_weights(self):</pre></div>\n</div>\n</ul>\n  <blockquote>AssertionError: 1 != 100\n</blockquote>\n</div>\n"}
\ No newline at end of file

devel/example_devel/instructor/cs108/unitgrade_data/Numpy-test_weights.json

-{"state": "pass", "run_id": 766225, "coverage_files_changed": null, "stdout": [[0, "Dashboard> Evaluation completed."]]}
\ No newline at end of file