Skip to content
Snippets Groups Projects
Commit d99a8e59 authored by tuhe's avatar tuhe
Browse files

Mosspy integration

parent 256509ac
Branches
No related tags found
No related merge requests found
Showing
with 73 additions and 472 deletions
......@@ -10,6 +10,7 @@ Unitgrade is an automatic report and exam evaluation framework that enables inst
- Students get a tamper-resistant file to create submissions which are uploaded
- Instructors can automatically verify the students solution using a Docker VM and run hidden tests
- Tests are quick to run and will integrate with your IDE
- Moss anti-plagiarism integration
**Note: This is the development version of unitgrade. If you are a student, please see http://gitlab.compute.dtu.dk/tuhe/unitgrade.**
......
File deleted
"""
Example student code. This file is automatically generated from the files in the instructor-directory
"""
def reverse_list(mylist):
"""
Given a list 'mylist' returns a list consisting of the same elements in reverse order. E.g.
......
"""
Example student code. This file is automatically generated from the files in the instructor-directory
"""
from src.unitgrade2.unitgrade2 import UTestCase, Report
from src.unitgrade2 import evaluate_report_student
from unitgrade2.unitgrade2 import UTestCase, Report, hide
from unitgrade2.unitgrade_helpers2 import evaluate_report_student
from cs103.homework1 import add
import cs103
class Week1(UTestCase):
""" The first question for week 1. """
def test_add(self):
from cs103.homework1 import add
self.assertEqualC(add(2,2))
self.assertEqualC(add(-100, 5))
@hide
def test_add_hidden(self):
# The @hide-decorator allows unitgrade to hide the test for students.
# See the output in the student directory.
self.assertEqualC(add(2,2))
class AutomaticPass(UTestCase):
def test_student_passed(self):
self.assertEqual(2,2)
@hide
def test_hidden_fail(self):
self.assertEqual(2,3)
import cs103
class Report3(Report):
title = "CS 101 Report 3"
questions = [(Week1, 20), (AutomaticPass, 10)] # Include a single question for 10 credits.
pack_imports = [cs103]
if __name__ == "__main__":
# from unitgrade_private2.hidden_gather_upload import gather_upload_to_campusnet
# gather_upload_to_campusnet(Report3())
evaluate_report_student(Report3())
# Unitgrade-private
# Unitgrade-devel
**Note: This is the development version of unitgrade. If you are a student, please see http://gitlab.compute.dtu.dk/tuhe/unitgrade.**
Unitgrade is an automatic report and exam evaluation framework that enables instructors to offer automatically evaluated programming assignments.
Unitgrade is build on pythons `unittest` framework so that the tests can be specified in a familiar syntax and will integrate with any modern IDE. What it offers beyond `unittest` is the ability to collect tests in reports (for automatic evaluation) and an easy and 100% safe mechanism for verifying the students results and creating additional, hidden tests. A powerful cache system allows instructors to automatically create test-answers based on a working solution.
- 100% Python `unittest` compatible
- No external configuration files: Just write a `unittest`
- No unnatural limitations: Use any package or framework. If you can `unittest` it, it works.
- Tests are quick to run and will integrate with your IDE
- Cache and hint-system makes tests easy to develop
- Granular security model:
- Students get public `unittests` for easy development of solutions
- Students get a tamper-resistant file to create submissions which are uploaded
- Students use a tamper-resistant file to create submissions which are uploaded
- Instructors can automatically verify the students solution using a Docker VM and run hidden tests
- Tests are quick to run and will integrate with your IDE
**Note: This is the development version of unitgrade. If you are a student, please see http://gitlab.compute.dtu.dk/tuhe/unitgrade.**
- Automatic Moss anti-plagiarism detection
- CMU Autolab integration (Experimental)
# Using unitgrade
The examples can be found in the `/examples/` directory: https://gitlab.compute.dtu.dk/tuhe/unitgrade_private/examples
......@@ -25,9 +28,9 @@ Although not required, it is recommended you maintain two version of the code:
- A fully-working version (i.e. all tests pass)
- A public version distributed to students (some code removed))
In this example, I will use `snipper` (see http://gitlab.compute.dtu.dk/tuhe/snipper) to synchronize the two versions automatically.
Let's look at an example. You need three files
```
I use `codesnipper` (see http://gitlab.compute.dtu.dk/tuhe/snipper) to synchronize the two versions automatically.
Let's look at an example. Suppose our course is called `cs101`, in which case we make three files in our private folder `instructor`:
```terminal
instructor/cs101/homework.py # This contains the students homework
instructor/cs101/report1.py # This contains the tests
instructor/cs101/deploy.py # A private file to deploy the tests
......@@ -123,7 +126,7 @@ if __name__ == "__main__":
- The second line set up the students directory (remember, we have included the solutions!) and remove the students solutions. You can check the results in the students folder.
### Using the framework as a student
You can now upload the `student' directory to the students. The students can run their tests either by running `cs101.report1` in their IDE or by typing:
You can now upload the `student` directory to the students. The students can run their tests either by running `cs101.report1` in their IDE or by typing:
```
python -m cs101.report1
```
......@@ -132,13 +135,12 @@ in the command line. This produces a detailed output of the test and the program
python -m cs101.report1_grade
```
This runs an identical set of tests, but produces a `.token` file the students can upload to get credit.
- The reason to have a seperate `report1_grade.py` script is to avoid accidential removal of tests.
- The `report1_grade.py` includes all tests and the main parts of the framework and is obfuscated by default. You can apply a much strong level of protection by using e.g. `pyarmor`.
- The `report1_token.token` file includes the outcome of the tests, the time taken, and all python source code in the package. In other words, the file can be used for manual grading, for plagirism detection and for detecting tampering.
- You can easily use the framework to include output of functions.
- See below for how to validate the students results
### How safe is this?
### How safe is Unitgrade?
Cheating within the framework is probably best done by manually editing the `.token`-file or by creating a broken set of tests. This involves risk of being trivially detected, for instance because tests have the wrong runtime, but more importantly
the framework automatically pack all the used source code and so if a student is cheating, there is no way to hide it for an instructor who looks at the results. If the
program is used in conjunction with automatic plagiarism software, cheating therefore involves both breaking the framework, and creating 'false' solutions which statistically match other students solutions, and then hope nobody bothers to check the output.
......
......@@ -5,6 +5,17 @@ if __name__ == "__main__":
import jinja2
data = {'bibtex': bib}
import glob
fls = [f for f in glob.glob("../examples/**/*.*", recursive=True) if f.split(".")[-1] in ["py", "txt", "shell"] ]
import os
# fls = [(f, ) for f in fls]
for file in fls:
with open(file, 'r') as f:
k = os.path.relpath(file, "../examples").replace(os.sep, "-").replace(".", "_")
data[k] = f.read()
with open("README.jinja.md", 'r') as f:
s = jinja2.Environment(loader=jinja2.FileSystemLoader([".", "../example"])).from_string(f.read()).render(data)
with open("../README.md", 'w') as f:
......
@online{unitgrade_devel,
title={Unitgrade-devel (0.0.1): \texttt{pip install unitgrade-devel}},
title={Unitgrade-devel (0.1.0): \texttt{pip install unitgrade-devel}},
url={https://lab.compute.dtu.dk/tuhe/unitgrade_private},
urldate = {2021-09-07},
urldate = {2021-09-08},
month={9},
publisher={Technical University of Denmark (DTU)},
author={Tue Herlau},
......
from testbook import testbook
nb = 'week02/Week_2_sol.ipynb'
import asyncio
import io, os, sys, types
# from IPython.nbformat import current
from nbformat import current
from IPython.core.interactiveshell import InteractiveShell
# from nbconvert import W
import asyncio
......@@ -97,28 +95,12 @@ class NotebookFinder(object):
sys.meta_path.append(NotebookFinder())
if __name__ == "__main__":
# test_func()
from unittest import TestCase
# if type(asyncio.get_event_loop_policy()) is WindowsProactorEventLoopPolicy:
# # close the existing proactor loop so if anyone has a handle on it they get an error instead of hanging.
# # there doesn't appear to be a way to ask if this has been accessed already, so this
# # will usually create a new loop and close it immediately
# proactor_loop = asyncio.get_event_loop()
# proactor_loop.close()
#
# # prefer the pre-3.8 default of Selector with add_reader support
# asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())
import nbformat
from nbconvert.preprocessors import ExecutePreprocessor
# from week02 import Week_2_sol
import importnb
file = "../../../example_jupyter/instructor/cs105/week2.ipynb"
file2 = 'week02/Week_2_sol.ipynb'
m = importnb.Notebook.load(file)
# importnb.Notebook.l
import nbclient
import nbformat
from nbparameterise import extract_parameters, replace_definitions, parameter_values
......@@ -138,7 +120,6 @@ if __name__ == "__main__":
# Execute the notebook with the new parameters
nbclient.execute(new_nb)
# with importnb.Notebook(file):
# import model
# import nbformat
......
File deleted
from report1intro import Report1Flat
from unitgrade_private2.hidden_create_files import setup_grade_file_report
from snipper import snip_dir
if __name__ == "__main__":
setup_grade_file_report(Report1Flat, minify=False, obfuscate=False, execute=False, with_coverage=True)
# from unitgrade_private2.hidden_gather_upload import gather_upload_to_campusnet
# gather_upload_to_campusnet((Report1Flat()))
# Deploy the files using snipper: https://gitlab.compute.dtu.dk/tuhe/snipper
snip_dir.snip_dir(source_dir="", dest_dir="../../students/programs", clean_destination_dir=True, exclude=['__pycache__', '*.token', 'deploy.py'])
import os
os.system("python report1intro_grade.py")
"""
from coverage import CoverageData
import coverage
cov2 = coverage.Coverage()
def setUp(self):
import trace
self.cov = cov2
self.cov.start()
# self.tracer.start()
# using obj_to_trace
def tearDown(self) -> None:
self.cov.stop()
print()
data = CoverageData()
# data.measured_files()
# data.lines()
data = self.cov.get_data()
# data.
for file in data.measured_files():
print(file)
print(data.lines(file))
print(data.arcs(file))
print( data.contexts_by_lineno(file))
# print(data[file])
- Idea: Measure coverage in setup/teardown. This gives a handful of covered lines.
- During setup, supply a dicionary to UTestCase of files, along with the lines that are removed.
- When running setup: Take the coverage report, and compare against files. Write functions/lines encountered to the cache dictionary. Rquires you to
- inspect the functions that are edited to figure out what is removed. This can probably be done by going upwars towards the first sensible class or function definition (which has not been removed).
- Supply a dictionary to UTestCase of files, along with the lines edited. Allow UTestCase to write this information to the
cache dictionary (i.e. lines removed). Then use this information when displaying helpful hints later.
"""
\ No newline at end of file
This diff is collapsed.
No preview for this file type
File added
from report1intro import Report1Flat
from unitgrade_private2.hidden_create_files import setup_grade_file_report
from snipper import snip_dir
if __name__ == "__main__":
setup_grade_file_report(Report1Flat, minify=False, obfuscate=False, execute=False, with_coverage=True)
# Deploy the files using snipper: https://gitlab.compute.dtu.dk/tuhe/snipper
snip_dir.snip_dir(source_dir="", dest_dir="../../students/week5", clean_destination_dir=True, exclude=['__pycache__', '*.token', 'deploy.py'])
# import os
# os.system("python report1intro_grade.py")
"""
Example student code. This file is automatically generated from the files in the instructor-directory
"""
from src.unitgrade2.unitgrade2 import Report, UTestCase, cache
from src.unitgrade2 import evaluate_report_student
from unitgrade2.unitgrade2 import Report, UTestCase, cache
from unitgrade2.unitgrade_helpers2 import evaluate_report_student
import numpy as np
import looping
from looping import bacteriaGrowth, clusterAnalysis, removeIncomplete, fermentationRate
......@@ -43,7 +40,7 @@ class Bacteria(UTestCase):
self.stest(100, 0.4, 1000, 99)
class ClusterAnalysis(UTestCase):
""" Test the cluster analysis method """
""" Cluster analysis """
def stest(self, n, seed):
np.random.seed(seed)
......@@ -105,7 +102,7 @@ class RemoveIncomplete(UTestCase):
class FermentationRate(UTestCase):
""" Test the fermentation rate question """
""" Fermentation rate """
def stest(self, x, lower, upper):
I = fermentationRate(x, lower, upper)
......@@ -137,6 +134,4 @@ class Report1Flat(Report):
pack_imports = [looping]
if __name__ == "__main__":
# Uncomment to simply run everything as a unittest:
# unittest.main(verbosity=2)
evaluate_report_student(Report1Flat())
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment