Skip to content
Snippets Groups Projects
Commit 7e820f22 authored by tuhe's avatar tuhe
Browse files

Updates

parent 8647ebf0
No related branches found
No related tags found
No related merge requests found
Showing
with 792 additions and 105 deletions
LICENSE 0 → 100644
Copyright (c) 2018 The Python Packaging Authority
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
\ No newline at end of file
include *.dat
......@@ -4,47 +4,35 @@ Unitgrade is an automatic report and exam evaluation framework that enables inst
Unitgrade is build on pythons `unittest` framework so that the tests can be specified in a familiar syntax and will integrate with any modern IDE. What it offers beyond `unittest` is the ability to collect tests in reports (for automatic evaluation) and an easy and 100% safe mechanism for verifying the students results and creating additional, hidden tests. A powerful cache system allows instructors to automatically create test-answers based on a working solution.
- 100% Python `unittest` compatible
- No external configuration files: Just write a `unittest`
- No unnatural limitations: Use any package or framework. If you can `unittest` it, it works.
- No external configuration files, just write a `unittest`
- No unnatural limitations: If you can `unittest` it, it works.
- Granular security model:
- Students get public `unittests` for easy development of solutions
- Students get a tamper-resistant file to create submissions which are uploaded
- Instructors can automatically verify the students solution using a Docker VM and run hidden tests
- Instructors can automatically verify the students solution using Docker VM and by running hidden tests
- Allow export of assignments to Autolab (no `make` file mysteries!)
- Tests are quick to run and will integrate with your IDE
## Installation
Unitgrade can be installed through pip using
Unitgrade can be installed using `pip`:
```
pip install git+https://git@gitlab.compute.dtu.dk/tuhe/unitgrade.git
pip install unitgrade
```
This will install unitgrade in your site-packages directory. If you want to upgrade an old installation of unitgrade:
```
pip install git+https://git@gitlab.compute.dtu.dk/tuhe/unitgrade.git --upgrade
pip install unitgrade --upgrade
```
If you are using anaconda+virtual environment you can install it as
```
source activate myenv
conda install git pip
pip install git+https://git@gitlab.compute.dtu.dk/tuhe/unitgrade.git
pip install unitgrade
```
Alternatively, simply use git-clone of the sources and add unitgrade to your python path.
When you are done, you should be able to import unitgrade:
```
import unitgrade
```
## Testing installation
I have provided an example project which illustrates all main features in a self-contained manner and which should
work immediately upon installation. The source can be found here: https://lab.compute.dtu.dk/tuhe/unitgrade/-/tree/master/cs101courseware_example
To run the example, first start a python console:
```
python
```
Then run the code
```
from cs101courseware_example import instructions
```
This will print on-screen instructions for how to use the system tailored to your user-specific installation path.
## Evaluating a report
Homework is broken down into **reports**. A report is a collection of questions which are individually scored, and each question may in turn involve multiple tests. Each report is therefore given an overall score based on a weighted average of how many tests are passed.
......@@ -83,12 +71,12 @@ To register your results, please run the file:
>>> cs101report1_grade.py
In the same manner as you ran this file.
```
Once you are happy with the result, run the alternative, not-easy-to-tamper-with script called `cs101report1_grade.py`:
Once you are happy with the result run the script with the `_grade.py`-postfix, in this case `cs101report1_grade.py`:
```
python cs101report1_grade.py
```
This runs the same tests, and generates a file `Report0_handin_18_of_18.token`. The file name indicates how many points you got. Upload this file to campusnet.
This runs the same tests, and generates a file `Report0_handin_18_of_18.token`. The file name indicates how many points you got. Upload this file to campusnet (and no other).
### Why are there two scripts?
The reason why we use a standard test script, and one with the `_grade.py` extension, is because the tests should both be easy to debug, but at the same time we have to prevent accidential changes to the test scripts. Hence, we include two versions of the tests.
......@@ -97,7 +85,7 @@ The reason why we use a standard test script, and one with the `_grade.py` exten
- **My non-grade script and the `_grade.py` script gives different number of points**
Since the two scripts should contain the same code, the reason is nearly certainly that you have made an (accidental) change to the test scripts. Please ensure both scripts are up-to-date and if the problem persists, try to get support.
- **Why is there a `*_resources_do_not_hand_in.dat` file? Should I also upload it?**
- **Why is there a `unitgrade` directory with a bunch of pickle files? Should I also upload them?**
No. The file contains the pre-computed test results your code is compared against. If you want to load this file manually, the unitgrade package contains helpful functions for doing so.
- **I am worried you might think I cheated because I opened the '_grade.py' script/token file**
......@@ -135,7 +123,7 @@ Yes.
That the script `report1_grade.py` is difficult to read is not the principle safety measure. Instead, it ensures there is no accidential tampering. If you muck around with these files and upload the result, we will very likely know.
- **I have private data on my computer. Will this be read or uploaded?**
No. The code will look for and upload your solutions, but it will not read/look at other directories in your computer. In the example provided with this code, this means you should expect unitgrade to read/run all files in the `cs101courseware_example`-directory, but **no** other files on your computer (unless some code in this directory load other files). So as long as you keep your private files out of the base courseware directory, you should be fine.
No. The code will look for and upload your solutions, but it will not read/look at other directories in your computer. In the example provided with this code, this means you should expect unitgrade to read/run all files in the `cs101courseware_example`-directory, but **no** other files on your computer. So as long as you keep your private files out of the base courseware directory, you should be fine.
- **Does this code install any spyware/etc.? Does it communicate with a website/online service?**
No. Unitgrade makes no changes outside the courseware directory and it does not do anything tricky. It reads/runs code and write the `.token` file.
......
......@@ -18,7 +18,7 @@ for d in os.listdir(wdir):
if "__" not in d and d != "instructions.py":
print("> ", d)
print("")
fprint("The file homework1.py is the file you edit as part of the course; you are welcome to open it and inspect the content, but right now it consists of some simple programming tasks plus instructions.")
fprint("The file looping.py is the file you edit as part of the course; you are welcome to open it and inspect the content, but right now it consists of some simple programming tasks plus instructions.")
fprint("The file cs101report1.py contains the actual tests of the program. All the tests are easily readable and the script will work with your debugger if you are using pycharm, however, we will run the script for the command line. ")
fprint("To do so, open a console, and change directory to the cs103 main directory using e.g.:")
tprint(f'cd "{wdir}"')
......
File added
File added
File deleted
[build-system]
requires = [
"setuptools>=42",
"wheel"
]
build-backend = "setuptools.build_meta"
\ No newline at end of file
from setuptools import setup
from unitgrade.version import __version__
setup(
name='unitgrade',
# Use this guide:
# https://packaging.python.org/tutorials/packaging-projects/
# from unitgrade2.version import __version__
import setuptools
with open("src/unitgrade2/version.py", "r", encoding="utf-8") as fh:
__version__ = fh.read().split(" = ")[1].strip()[1:-1]
# long_description = fh.read()
with open("README.md", "r", encoding="utf-8") as fh:
long_description = fh.read()
setuptools.setup(
name="unitgrade",
version=__version__,
packages=['unitgrade', 'cs101courseware_example'],
author="Tue Herlau",
author_email="tuhe@dtu.dk",
description="A student homework/exam evaluation framework build on pythons unittest framework.",
long_description=long_description,
long_description_content_type="text/markdown",
url='https://lab.compute.dtu.dk/tuhe/unitgrade',
license='Apache',
author='Tue Herlau',
author_email='tuhe@dtu.dk',
description='A lightweight student evaluation framework build on unittest',
include_package_data=True,
install_requires=['numpy', 'jinja2', 'tabulate', 'sklearn', 'compress_pickle', "pyfiglet"],
project_urls={
"Bug Tracker": "https://lab.compute.dtu.dk/tuhe/unitgrade/issues",
},
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
],
package_dir={"": "src"},
packages=setuptools.find_packages(where="src"),
python_requires=">=3.8",
license="MIT",
install_requires=['numpy', 'tabulate', 'tqdm', "pyfiglet", "colorama", "coverage"],
)
from unitgrade2.version import __version__
import os
# DONT't import stuff here since install script requires __version__
......@@ -34,4 +33,4 @@ def cache_read(file_name):
else:
return None
from unitgrade2.unitgrade2 import Hidden, myround, mfloor, msum, Capturing, ActiveProgress
from unitgrade2.unitgrade2 import myround, mfloor, msum, Capturing, ActiveProgress
File added
File added
"""
git add . && git commit -m "Options" && git push && pip install git+ssh://git@gitlab.compute.dtu.dk/tuhe/unitgrade.git --upgrade
"""
# from . import cache_read
import unittest
import numpy as np
import sys
from io import StringIO
import collections
import re
import threading
import tqdm
import time
import pickle
import itertools
import os
from io import StringIO
import io
from unittest.runner import _WritelnDecorator
from typing import Any
import inspect
import textwrap
import colorama
from colorama import Fore
from functools import _make_key, RLock
from collections import namedtuple
import unittest
import time
_CacheInfo = namedtuple("CacheInfo", ["hits", "misses", "maxsize", "currsize"])
colorama.init(autoreset=True) # auto resets your settings after every output
def gprint(s):
print(f"{Fore.GREEN}{s}")
myround = lambda x: np.round(x) # required.
msum = lambda x: sum(x)
mfloor = lambda x: np.floor(x)
def setup_dir_by_class(C, base_dir):
name = C.__class__.__name__
# base_dir = os.path.join(base_dir, name)
# if not os.path.isdir(base_dir):
# os.makedirs(base_dir)
return base_dir, name
class Hidden:
def hide(self):
return True
class Logger(object):
def __init__(self, buffer):
assert False
self.terminal = sys.stdout
self.log = buffer
......@@ -44,6 +52,7 @@ class Logger(object):
# this flush method is needed for python 3 compatibility.
pass
class Capturing(list):
def __init__(self, *args, stdout=None, unmute=False, **kwargs):
self._stdout = stdout
......@@ -71,6 +80,7 @@ class Capturing(list):
if self.capture_errors:
sys.sterr = self._sterr
class Capturing2(Capturing):
def __exit__(self, *args):
lines = self._stringio.getvalue().splitlines()
......@@ -86,227 +96,27 @@ class Capturing2(Capturing):
self.numbers = numbers
class QItem(unittest.TestCase):
title = None
testfun = None
tol = 0
estimated_time = 0.42
_precomputed_payload = None
_computed_answer = None # Internal helper to later get results.
weight = 1 # the weight of the question.
def __init__(self, question=None, *args, **kwargs):
if self.tol > 0 and self.testfun is None:
self.testfun = self.assertL2Relative
elif self.testfun is None:
self.testfun = self.assertEqual
self.name = self.__class__.__name__
# self._correct_answer_payload = correct_answer_payload
self.question = question
super().__init__(*args, **kwargs)
if self.title is None:
self.title = self.name
def _safe_get_title(self):
if self._precomputed_title is not None:
return self._precomputed_title
return self.title
def assertNorm(self, computed, expected, tol=None):
if tol == None:
tol = self.tol
diff = np.abs( (np.asarray(computed).flat- np.asarray(expected)).flat )
nrm = np.sqrt(np.sum( diff ** 2))
self.error_computed = nrm
if nrm > tol:
print(f"Not equal within tolerance {tol}; norm of difference was {nrm}")
print(f"Element-wise differences {diff.tolist()}")
self.assertEqual(computed, expected, msg=f"Not equal within tolerance {tol}")
def assertL2(self, computed, expected, tol=None):
if tol == None:
tol = self.tol
diff = np.abs( (np.asarray(computed) - np.asarray(expected)) )
self.error_computed = np.max(diff)
if np.max(diff) > tol:
print(f"Not equal within tolerance {tol=}; deviation was {np.max(diff)=}")
print(f"Element-wise differences {diff.tolist()}")
self.assertEqual(computed, expected, msg=f"Not equal within tolerance {tol=}, {np.max(diff)=}")
def assertL2Relative(self, computed, expected, tol=None):
if tol == None:
tol = self.tol
diff = np.abs( (np.asarray(computed) - np.asarray(expected)) )
diff = diff / (1e-8 + np.abs( (np.asarray(computed) + np.asarray(expected)) ) )
self.error_computed = np.max(np.abs(diff))
if np.sum(diff > tol) > 0:
print(f"Not equal within tolerance {tol}")
print(f"Element-wise differences {diff.tolist()}")
self.assertEqual(computed, expected, msg=f"Not equal within tolerance {tol}")
def precomputed_payload(self):
return self._precomputed_payload
def precompute_payload(self):
# Pre-compute resources to include in tests (useful for getting around rng).
pass
def compute_answer(self, unmute=False):
raise NotImplementedError("test code here")
def test(self, computed, expected):
self.testfun(computed, expected)
def get_points(self, verbose=False, show_expected=False, show_computed=False,unmute=False, passall=False, silent=False, **kwargs):
possible = 1
computed = None
def show_computed_(computed):
print(">>> Your output:")
print(computed)
def show_expected_(expected):
print(">>> Expected output (note: may have been processed; read text script):")
print(expected)
correct = self._correct_answer_payload
try:
if unmute: # Required to not mix together print stuff.
print("")
computed = self.compute_answer(unmute=unmute)
except Exception as e:
if not passall:
if not silent:
print("\n=================================================================================")
print(f"When trying to run test class '{self.name}' your code threw an error:", e)
show_expected_(correct)
import traceback
print(traceback.format_exc())
print("=================================================================================")
return (0, possible)
if self._computed_answer is None:
self._computed_answer = computed
if show_expected or show_computed:
print("\n")
if show_expected:
show_expected_(correct)
if show_computed:
show_computed_(computed)
try:
if not passall:
self.test(computed=computed, expected=correct)
except Exception as e:
if not silent:
print("\n=================================================================================")
print(f"Test output from test class '{self.name}' does not match expected result. Test error:")
print(e)
show_computed_(computed)
show_expected_(correct)
return (0, possible)
return (1, possible)
def score(self):
try:
self.test()
except Exception as e:
return 0
return 1
class QPrintItem(QItem):
def compute_answer_print(self):
"""
Generate output which is to be tested. By default, both text written to the terminal using print(...) as well as return values
are send to process_output (see compute_answer below). In other words, the text generated is:
res = compute_Answer_print()
txt = (any terminal output generated above)
numbers = (any numbers found in terminal-output txt)
self.test(process_output(res, txt, numbers), <expected result>)
:return: Optional values for comparison
"""
raise Exception("Generate output here. The output is passed to self.process_output")
def process_output(self, res, txt, numbers):
return res
def compute_answer(self, unmute=False):
with Capturing(unmute=unmute) as output:
res = self.compute_answer_print()
s = "\n".join(output)
s = rm_progress_bar(s) # Remove progress bar.
numbers = extract_numbers(s)
self._computed_answer = (res, s, numbers)
return self.process_output(res, s, numbers)
class OrderedClassMembers(type):
@classmethod
def __prepare__(self, name, bases):
return collections.OrderedDict()
def __new__(self, name, bases, classdict):
ks = list(classdict.keys())
for b in bases:
ks += b.__ordered__
classdict['__ordered__'] = [key for key in ks if key not in ('__module__', '__qualname__')]
return type.__new__(self, name, bases, classdict)
class QuestionGroup(metaclass=OrderedClassMembers):
title = "Untitled question"
partially_scored = False
t_init = 0 # Time spend on initialization (placeholder; set this externally).
estimated_time = 0.42
has_called_init_ = False
_name = None
_items = None
@property
def items(self):
if self._items == None:
self._items = []
members = [gt for gt in [getattr(self, gt) for gt in self.__ordered__ if gt not in ["__classcell__", "__init__"]] if inspect.isclass(gt) and issubclass(gt, QItem)]
for I in members:
self._items.append( I(question=self))
return self._items
@items.setter
def items(self, value):
self._items = value
@property
def name(self):
if self._name == None:
self._name = self.__class__.__name__
return self._name #
@name.setter
def name(self, val):
self._name = val
def init(self):
# Can be used to set resources relevant for this question instance.
pass
def init_all_item_questions(self):
for item in self.items:
if not item.question.has_called_init_:
item.question.init()
item.question.has_called_init_ = True
# @classmethod
# class OrderedClassMembers(type):
# def __prepare__(self, name, bases):
# assert False
# return collections.OrderedDict()
#
# def __new__(self, name, bases, classdict):
# ks = list(classdict.keys())
# for b in bases:
# ks += b.__ordered__
# classdict['__ordered__'] = [key for key in ks if key not in ('__module__', '__qualname__')]
# return type.__new__(self, name, bases, classdict)
class Report():
class Report:
title = "report title"
version = None
questions = []
pack_imports = []
individual_imports = []
nL = 80 # Maximum line width
nL = 120 # Maximum line width
@classmethod
def reset(cls):
......@@ -322,7 +132,11 @@ class Report():
return inspect.getfile(type(self))
def _import_base_relative(self):
if hasattr(self.pack_imports[0], '__path__'):
root_dir = self.pack_imports[0].__path__._path[0]
else:
root_dir = self.pack_imports[0].__file__
root_dir = os.path.dirname(root_dir)
relative_path = os.path.relpath(self._file(), root_dir)
modules = os.path.normpath(relative_path[:-3]).split(os.sep)
......@@ -330,7 +144,6 @@ class Report():
def __init__(self, strict=False, payload=None):
working_directory = os.path.abspath(os.path.dirname(self._file()))
self.wdir, self.name = setup_dir_by_class(self, working_directory)
# self.computed_answers_file = os.path.join(self.wdir, self.name + "_resources_do_not_hand_in.dat")
for (q, _) in self.questions:
......@@ -338,44 +151,44 @@ class Report():
if payload is not None:
self.set_payload(payload, strict=strict)
# else:
# if os.path.isfile(self.computed_answers_file):
# self.set_payload(cache_read(self.computed_answers_file), strict=strict)
# else:
# s = f"> Warning: The pre-computed answer file, {os.path.abspath(self.computed_answers_file)} is missing. The framework will NOT work as intended. Reasons may be a broken local installation."
# if strict:
# raise Exception(s)
# else:
# print(s)
def main(self, verbosity=1):
# Run all tests using standard unittest (nothing fancy).
import unittest
loader = unittest.TestLoader()
for q, _ in self.questions:
import time
start = time.time() # A good proxy for setup time is to
suite = loader.loadTestsFromTestCase(q)
unittest.TextTestRunner(verbosity=verbosity).run(suite)
total = time.time() - start
q.time = total
def _setup_answers(self):
def _setup_answers(self, with_coverage=False):
if with_coverage:
for q, _ in self.questions:
q._with_coverage = True
q._report = self
self.main() # Run all tests in class just to get that out of the way...
report_cache = {}
for q, _ in self.questions:
# print(self.questions)
if hasattr(q, '_save_cache'):
q()._save_cache()
q._cache['time'] = q.time
report_cache[q.__qualname__] = q._cache
print("q is", q())
q()._cache_put('time', q.time) # = q.time
report_cache[q.__qualname__] = q._cache2
else:
report_cache[q.__qualname__] = {'no cache see _setup_answers in unitgrade2.py': True}
if with_coverage:
for q, _ in self.questions:
q._with_coverage = False
return report_cache
def set_payload(self, payloads, strict=False):
for q, _ in self.questions:
q._cache = payloads[q.__qualname__]
def rm_progress_bar(txt):
# More robust version. Apparently length of bar can depend on various factors, so check for order of symbols.
nlines = []
......@@ -390,9 +203,10 @@ def rm_progress_bar(txt):
nlines.append(l)
return "\n".join(nlines)
def extract_numbers(txt):
# txt = rm_progress_bar(txt)
numeric_const_pattern = '[-+]? (?: (?: \d* \. \d+ ) | (?: \d+ \.? ) )(?: [Ee] [+-]? \d+ ) ?'
numeric_const_pattern = r'[-+]? (?: (?: \d* \. \d+ ) | (?: \d+ \.? ) )(?: [Ee] [+-]? \d+ ) ?'
rx = re.compile(numeric_const_pattern, re.VERBOSE)
all = rx.findall(txt)
all = [float(a) if ('.' in a or "e" in a) else int(a) for a in all]
......@@ -401,16 +215,20 @@ def extract_numbers(txt):
raise Exception("unitgrade.unitgrade.py: Warning, too many numbers!", len(all))
return all
class ActiveProgress():
def __init__(self, t, start=True, title="my progress bar",show_progress_bar=True):
def __init__(self, t, start=True, title="my progress bar", show_progress_bar=True, file=None):
if file == None:
file = sys.stdout
self.file = file
self.t = t
self._running = False
self.title = title
self.dt = 0.1
self.dt = 0.01
self.n = int(np.round(self.t / self.dt))
self.show_progress_bar = show_progress_bar
self.pbar = None
# self.pbar = tqdm.tqdm(total=self.n)
if start:
self.start()
......@@ -427,17 +245,17 @@ class ActiveProgress():
self._running = False
if self.show_progress_bar:
self.thread.join()
if hasattr(self, 'pbar') and self.pbar is not None:
if self.pbar is not None:
self.pbar.update(1)
self.pbar.close()
self.pbar = None
sys.stdout.flush()
self.file.flush()
return time.time() - self.time_started
def run(self):
self.pbar = tqdm.tqdm(total=self.n, file=sys.stdout, position=0, leave=False, desc=self.title, ncols=100,
bar_format='{l_bar}{bar}| [{elapsed}<{remaining}]') # , unit_scale=dt, unit='seconds'):
self.pbar = tqdm.tqdm(total=self.n, file=self.file, position=0, leave=False, desc=self.title, ncols=100,
bar_format='{l_bar}{bar}| [{elapsed}<{remaining}]')
for _ in range(self.n - 1): # Don't terminate completely; leave bar at 99% done until terminate.
if not self._running:
......@@ -448,111 +266,34 @@ class ActiveProgress():
time.sleep(self.dt)
self.pbar.update(1)
def dprint(first, last, nL, extra = "", file=None, dotsym='.', color='white'):
if file == None:
file = sys.stdout
# ss = self.item_title_print
# state = "PASS" if success else "FAILED"
dot_parts = (dotsym * max(0, nL - len(last) - len(first)))
# if self.show_progress_bar or True:
print(first + dot_parts, end="", file=file)
# else:
# print(dot_parts, end="", file=self.cc.file)
last += extra
# if tsecs >= 0.5:
# state += " (" + str(tsecs) + " seconds)"
print(last, file=file)
from unittest.suite import _isnotsuite
# class MySuite(unittest.suite.TestSuite): # Not sure we need this one anymore.
# raise Exception("no suite")
# pass
def instance_call_stack(instance):
s = "-".join(map(lambda x: x.__name__, instance.__class__.mro()))
return s
def get_class_that_defined_method(meth):
for cls in inspect.getmro(meth.im_class):
if meth.__name__ in cls.__dict__:
return cls
return None
def caller_name(skip=2):
"""Get a name of a caller in the format module.class.method
`skip` specifies how many levels of stack to skip while getting caller
name. skip=1 means "who calls me", skip=2 "who calls my caller" etc.
An empty string is returned if skipped levels exceed stack height
"""
stack = inspect.stack()
start = 0 + skip
if len(stack) < start + 1:
return ''
parentframe = stack[start][0]
name = []
module = inspect.getmodule(parentframe)
# `modname` can be None when frame is executed directly in console
# TODO(techtonik): consider using __main__
if module:
name.append(module.__name__)
# detect classname
if 'self' in parentframe.f_locals:
# I don't know any way to detect call from the object method
# XXX: there seems to be no way to detect static method call - it will
# be just a function call
name.append(parentframe.f_locals['self'].__class__.__name__)
codename = parentframe.f_code.co_name
if codename != '<module>': # top level usually
name.append( codename ) # function or a method
## Avoid circular refs and frame leaks
# https://docs.python.org/2.7/library/inspect.html#the-interpreter-stack
del parentframe, stack
return ".".join(name)
def get_class_from_frame(fr):
import inspect
args, _, _, value_dict = inspect.getargvalues(fr)
# we check the first parameter for the frame function is
# named 'self'
if len(args) and args[0] == 'self':
# in that case, 'self' will be referenced in value_dict
instance = value_dict.get('self', None)
if instance:
# return its class
# isinstance(instance, Testing) # is the actual class instance.
return getattr(instance, '__class__', None)
# return None otherwise
return None
from typing import Any
import inspect, gc
def giveupthefunc():
frame = inspect.currentframe()
code = frame.f_code
globs = frame.f_globals
functype = type(lambda: 0)
funcs = []
for func in gc.get_referrers(code):
if type(func) is functype:
if getattr(func, "__code__", None) is code:
if getattr(func, "__globals__", None) is globs:
funcs.append(func)
if len(funcs) > 1:
return None
return funcs[0] if funcs else None
from collections import defaultdict
class UTextResult(unittest.TextTestResult):
nL = 80
number = -1 # HAcky way to set question number.
show_progress_bar = True
cc = None
def __init__(self, stream, descriptions, verbosity):
super().__init__(stream, descriptions, verbosity)
self.successes = []
def printErrors(self) -> None:
# if self.dots or self.showAll:
# self.stream.writeln()
# if hasattr(self, 'cc'):
# self.cc.terminate()
# self.cc_terminate(success=False)
self.printErrorList('ERROR', self.errors)
self.printErrorList('FAIL', self.failures)
......@@ -563,69 +304,51 @@ class UTextResult(unittest.TextTestResult):
def addFailure(self, test, err):
super(unittest.TextTestResult, self).addFailure(test, err)
self.cc_terminate(success=False)
# if self.showAll:
# self.stream.writeln("FAIL")
# elif self.dots:
# self.stream.write('F')
# self.stream.flush()
def addSuccess(self, test: unittest.case.TestCase) -> None:
# super().addSuccess(test)
self.successes.append(test)
# super().addSuccess(test)
# hidden = issubclass(item.__class__, Hidden)
# # if not hidden:
# # print(ss, end="")
# # sys.stdout.flush()
# start = time.time()
#
# (current, possible) = item.get_points(show_expected=show_expected, show_computed=show_computed,unmute=unmute, passall=passall, silent=silent)
# q_[j] = {'w': item.weight, 'possible': possible, 'obtained': current, 'hidden': hidden, 'computed': str(item._computed_answer), 'title': item.title}
# tsecs = np.round(time.time()-start, 2)
self.cc_terminate()
def cc_terminate(self, success=True):
if self.show_progress_bar or True:
tsecs = np.round(self.cc.terminate(), 2)
sys.stdout.flush()
self.cc.file.flush()
ss = self.item_title_print
print(self.item_title_print + ('.' * max(0, self.nL - 4 - len(ss))), end="")
# current = 1
# possible = 1
# current == possible
ss = "PASS" if success else "FAILED"
if tsecs >= 0.1:
ss += " (" + str(tsecs) + " seconds)"
print(ss)
state = "PASS" if success else "FAILED"
dot_parts = ('.' * max(0, self.nL - len(state) - len(ss)))
if self.show_progress_bar or True:
print(self.item_title_print + dot_parts, end="", file=self.cc.file)
else:
print(dot_parts, end="", file=self.cc.file)
if tsecs >= 0.5:
state += " (" + str(tsecs) + " seconds)"
print(state, file=self.cc.file)
def startTest(self, test):
# super().startTest(test)
j =self.testsRun
# j =self.testsRun
self.testsRun += 1
# print("Starting the test...")
# show_progress_bar = True
n = UTextResult.number
item_title = self.getDescription(test)
# item_title = item_title.split("\n")[0]
# item_title = self.getDescription(test)
item_title = test.shortDescription() # Better for printing (get from cache).
if item_title == None:
# For unittest framework where getDescription may return None.
item_title = self.getDescription(test)
# test.countTestCases()
self.item_title_print = "*** q%i.%i) %s" % (n + 1, j + 1, item_title)
self.item_title_print = " * q%i.%i) %s" % (UTextResult.number + 1, self.testsRun, item_title)
estimated_time = 10
nL = 80
#
if self.show_progress_bar or True:
self.cc = ActiveProgress(t=estimated_time, title=self.item_title_print, show_progress_bar=self.show_progress_bar)
self.cc = ActiveProgress(t=estimated_time, title=self.item_title_print, show_progress_bar=self.show_progress_bar, file=sys.stdout)
else:
print(self.item_title_print + ('.' * max(0, nL - 4 - len(self.item_title_print))), end="")
print(self.item_title_print + ('.' * max(0, self.nL - 4 - len(self.item_title_print))), end="")
self._test = test
self._stdout = sys.stdout
sys.stdout = io.StringIO()
def stopTest(self, test):
sys.stdout = self._stdout
super().stopTest(test)
def _setupStdout(self):
if self._previousTestClass == None:
......@@ -635,30 +358,22 @@ class UTextResult(unittest.TextTestResult):
else:
q_title_print = "<unnamed test. See unitgrade.py>"
# q_title_print = "some printed title..."
cc = ActiveProgress(t=total_estimated_time, title=q_title_print, show_progress_bar=self.show_progress_bar)
self.cc = cc
def _restoreStdout(self): # Used when setting up the test.
if self._previousTestClass == None:
if self._previousTestClass is None:
q_time = self.cc.terminate()
q_time = np.round(q_time, 2)
sys.stdout.flush()
if self.show_progress_bar:
print(self.cc.title, end="")
# start = 10
# q_time = np.round(time.time() - start, 2)
nL = 80
print(" " * max(0, nL - len(self.cc.title)) + (
" (" + str(q_time) + " seconds)" if q_time >= 0.1 else "")) # if q.name in report.payloads else "")
# print("=" * nL)
print(" " * max(0, self.nL - len(self.cc.title)) + (" (" + str(q_time) + " seconds)" if q_time >= 0.5 else ""))
from unittest.runner import _WritelnDecorator
from io import StringIO
class UTextTestRunner(unittest.TextTestRunner):
def __init__(self, *args, **kwargs):
from io import StringIO
stream = StringIO()
stream = io.StringIO()
super().__init__(*args, stream=stream, **kwargs)
def _makeResult(self):
......@@ -667,17 +382,6 @@ class UTextTestRunner(unittest.TextTestRunner):
stream = _WritelnDecorator(stream)
return self.resultclass(stream, self.descriptions, self.verbosity)
def wrapper(foo):
def magic(self):
s = "-".join(map(lambda x: x.__name__, self.__class__.mro()))
# print(s)
foo(self)
magic.__doc__ = foo.__doc__
return magic
from functools import update_wrapper, _make_key, RLock
from collections import namedtuple
_CacheInfo = namedtuple("CacheInfo", ["hits", "misses", "maxsize", "currsize"])
def cache(foo, typed=False):
""" Magic cache wrapper
......@@ -686,30 +390,54 @@ def cache(foo, typed=False):
maxsize = None
def wrapper(self, *args, **kwargs):
key = (self.cache_id(), ("@cache", foo.__name__, _make_key(args, kwargs, typed)))
# key = (self.cache_id(), '@cache')
# if self._cache_contains[key]
if not self._cache_contains(key):
value = foo(self, *args, **kwargs)
self._cache_put(key, value)
else:
value = self._cache_get(key)
return value
return wrapper
def get_hints(ss):
if ss == None:
return None
try:
ss = textwrap.dedent(ss)
ss = ss.replace('''"""''', "").strip()
hints = ["hints:", ]
j = np.argmax([ss.lower().find(h) for h in hints])
h = hints[j]
ss = ss[ss.find(h) + len(h) + 1:]
ss = "\n".join([l for l in ss.split("\n") if not l.strip().startswith(":")])
ss = textwrap.dedent(ss)
ss = ss.strip()
return ss
except Exception as e:
print("bad hints", ss, e)
class UTestCase(unittest.TestCase):
_outcome = None # A dictionary which stores the user-computed outcomes of all the tests. This differs from the cache.
_cache = None # Read-only cache. Ensures method always produce same result.
_cache2 = None # User-written cache.
_with_coverage = False
_report = None # The report used. This is very, very hacky and should always be None. Don't rely on it!
def capture(self):
return Capturing2(stdout=self._stdout)
if hasattr(self, '_stdout') and self._stdout is not None:
file = self._stdout
else:
# self._stdout = sys.stdout
# sys._stdout = io.StringIO()
file = sys.stdout
return Capturing2(stdout=file)
@classmethod
def question_title(cls):
""" Return the question title """
return cls.__doc__.strip().splitlines()[0].strip() if cls.__doc__ != None else cls.__qualname__
return cls.__doc__.strip().splitlines()[0].strip() if cls.__doc__ is not None else cls.__qualname__
@classmethod
def reset(cls):
......@@ -719,28 +447,64 @@ class UTestCase(unittest.TestCase):
cls._cache2 = None
def _callSetUp(self):
self._stdout = sys.stdout
import io
sys.stdout = io.StringIO()
super().setUp()
# print("Setting up...")
if self._with_coverage:
if not hasattr(self._report, 'covcache'):
self._report.covcache = {}
import coverage
self.cov = coverage.Coverage()
self.cov.start()
self.setUp()
def _callTearDown(self):
sys.stdout = self._stdout
super().tearDown()
# print("asdfsfd")
self.tearDown()
if self._with_coverage:
from pathlib import Path
from snipper import snipper
self.cov.stop()
data = self.cov.get_data()
base, _, _ = self._report._import_base_relative()
for file in data.measured_files():
file = os.path.normpath(file)
root = Path(base)
child = Path(file)
if root in child.parents:
with open(child, 'r') as f:
s = f.read()
lines = s.splitlines()
garb = 'GARBAGE'
lines2 = snipper.censor_code(lines, keep=True)
assert len(lines) == len(lines2)
for l in data.contexts_by_lineno(file):
if lines2[l].strip() == garb:
if self.cache_id() not in self._report.covcache:
self._report.covcache[self.cache_id()] = {}
rel = os.path.relpath(child, root)
cc = self._report.covcache[self.cache_id()]
j = 0
for j in range(l, -1, -1):
if "def" in lines2[j] or "class" in lines2[j]:
break
from snipper.snipper import gcoms
fun = lines2[j]
comments, _ = gcoms("\n".join(lines2[j:l]))
if rel not in cc:
cc[rel] = {}
cc[rel][fun] = (l, "\n".join(comments))
self._cache_put((self.cache_id(), 'coverage'), self._report.covcache)
def shortDescriptionStandard(self):
sd = super().shortDescription()
if sd == None:
if sd is None:
sd = self._testMethodName
return sd
def shortDescription(self):
# self._testMethodDoc.strip().splitlines()[0].strip()
sd = self.shortDescriptionStandard()
title = self._cache_get((self.cache_id(), 'title'), sd)
return title if title != None else sd
return title if title is not None else sd
@property
def title(self):
......@@ -751,37 +515,31 @@ class UTestCase(unittest.TestCase):
self._cache_put((self.cache_id(), 'title'), value)
def _get_outcome(self):
if not (self.__class__, '_outcome') or self.__class__._outcome == None:
if not (self.__class__, '_outcome') or self.__class__._outcome is None:
self.__class__._outcome = {}
return self.__class__._outcome
def _callTestMethod(self, testMethod):
t = time.time()
self._ensure_cache_exists() # Make sure cache is there.
if self._testMethodDoc != None:
# Ensure the cache is eventually updated with the right docstring.
if self._testMethodDoc is not None:
self._cache_put((self.cache_id(), 'title'), self.shortDescriptionStandard())
# Fix temp cache here (for using the @cache decorator)
self._cache2[ (self.cache_id(), 'assert') ] = {}
self._cache2[(self.cache_id(), 'assert')] = {}
res = testMethod()
elapsed = time.time() - t
# self._cache_put( (self.cache_id(), 'title'), self.shortDescription() )
self._get_outcome()[self.cache_id()] = res
self._cache_put((self.cache_id(), "time"), elapsed)
# This is my base test class. So what is new about it?
def cache_id(self):
c = self.__class__.__qualname__
m = self._testMethodName
return (c,m)
return c, m
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._load_cache()
self._assert_cache_index = 0
# self.cache_indexes = defaultdict(lambda: 0)
def _ensure_cache_exists(self):
if not hasattr(self.__class__, '_cache') or self.__class__._cache == None:
......@@ -802,18 +560,23 @@ class UTestCase(unittest.TestCase):
return key in self.__class__._cache
def wrap_assert(self, assert_fun, first, *args, **kwargs):
# sys.stdout = self._stdout
key = (self.cache_id(), 'assert')
if not self._cache_contains(key):
print("Warning, framework missing", key)
cache = self._cache_get(key, {})
self.__class__._cache[
key] = {} # A new dict. We manually insert it because we have to use that the dict is mutable.
cache = self._cache_get(key)
id = self._assert_cache_index
if not id in cache:
print("Warning, framework missing cache index", key, "id =", id)
_expected = cache.get(id, first)
assert_fun(first, _expected, *args, **kwargs)
_expected = cache.get(id, f"Key {id} not found in cache; framework files missing. Please run deploy()")
# The order of these calls is important. If the method assert fails, we should still store the correct result in cache.
cache[id] = first
self._cache_put(key, cache)
self._assert_cache_index += 1
assert_fun(first, _expected, *args, **kwargs)
def assertEqualC(self, first: Any, msg: Any = ...) -> None:
self.wrap_assert(self.assertEqual, first, msg)
......@@ -833,27 +596,79 @@ class UTestCase(unittest.TestCase):
# But you can also set cache explicitly.
def _load_cache(self):
if self._cache != None: # Cache already loaded. We will not load it twice.
if self._cache is not None: # Cache already loaded. We will not load it twice.
return
# raise Exception("Loaded cache which was already set. What is going on?!")
cfile = self._cache_file()
# print("Loading cache from", cfile)
if os.path.exists(cfile):
try:
with open(cfile, 'rb') as f:
data = pickle.load(f)
self.__class__._cache = data
except Exception as e:
print("Bad cache", cfile)
print(e)
else:
print("Warning! data file not found", cfile)
def _feedErrorsToResult(self, result, errors):
""" Use this to show hints on test failure. """
if not isinstance(result, UTextResult):
er = [e for e, v in errors if v != None]
if len(er) > 0:
hints = []
key = (self.cache_id(), 'coverage')
if self._cache_contains(key):
CC = self._cache_get(key)
for id in CC:
if id == self.cache_id():
cl, m = id
gprint(f"> An error occured while solving: {cl}.{m}. The files/methods you need to edit are:") # For the test {id} in {file} you should edit:")
for file in CC[id]:
rec = CC[id][file]
gprint(f"> * {file}")
for l in rec:
_, comments = CC[id][file][l]
hint = get_hints(comments)
if hint != None:
hints.append(hint)
gprint(f"> - {l}")
er = er[0]
doc = er._testMethodDoc
if doc is not None:
hint = get_hints(er._testMethodDoc)
if hint is not None:
hints = [hint] + hints
if len(hints) > 0:
gprint("> Hints:")
gprint(textwrap.indent("\n".join(hints), "> "))
super()._feedErrorsToResult(result, errors)
def startTestRun(self):
# print("asdfsdaf 11", file=sys.stderr)
super().startTestRun()
# print("asdfsdaf")
def _callTestMethod(self, method):
# print("asdfsdaf")
super()._callTestMethod(method)
def hide(func):
return func
def makeRegisteringDecorator(foreignDecorator):
"""
Returns a copy of foreignDecorator, which is identical in every
way(*), except also appends a .decorator property to the callable it
spits out.
"""
def newDecorator(func):
# Call to newDecorator(method)
# Exactly like old decorator, but output keeps track of what decorated it
......@@ -864,7 +679,6 @@ def makeRegisteringDecorator(foreignDecorator):
newDecorator.__name__ = foreignDecorator.__name__
newDecorator.__doc__ = foreignDecorator.__doc__
# (*)We can be somewhat "hygienic", but newDecorator still isn't signature-preserving, i.e. you will not be able to get a runtime list of parameters. For that, you need hackish libraries...but in this case, the only argument is func, so it's not a big issue
return newDecorator
hide = makeRegisteringDecorator(hide)
......@@ -888,4 +702,4 @@ def methodsWithDecorator(cls, decorator):
if maybeDecorated.decorator == decorator:
print(maybeDecorated)
yield maybeDecorated
# 817
......@@ -2,19 +2,13 @@ import numpy as np
from tabulate import tabulate
from datetime import datetime
import pyfiglet
from unitgrade2 import Hidden, myround, msum, mfloor, ActiveProgress
from unitgrade2 import __version__
from unitgrade2 import msum
import unittest
# from unitgrade2.unitgrade2 import MySuite
from unitgrade2.unitgrade2 import UTextResult
import inspect
import os
import argparse
import sys
import time
import threading # don't import Thread bc. of minify issue.
import tqdm # don't do from tqdm import tqdm because of minify-issue
parser = argparse.ArgumentParser(description='Evaluate your report.', epilog="""Example:
To run all tests in a report:
......@@ -109,31 +103,27 @@ def evaluate_report(report, question=None, qitem=None, passall=False, verbose=Fa
show_tol_err=False,
big_header=True):
from unitgrade2.version import __version__
from src.unitgrade2.version import __version__
now = datetime.now()
if big_header:
ascii_banner = pyfiglet.figlet_format("UnitGrade", font="doom")
b = "\n".join( [l for l in ascii_banner.splitlines() if len(l.strip()) > 0] )
else:
b = "Unitgrade"
print(b + " v" + __version__)
dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
print("Started: " + dt_string)
print(b + " v" + __version__ + ", started: " + dt_string+ "\n")
# print("Started: " + dt_string)
s = report.title
if hasattr(report, "version") and report.version is not None:
s += " version " + report.version
print("Evaluating " + s, "(use --help for options)" if show_help_flag else "")
print(s, "(use --help for options)" if show_help_flag else "")
# print(f"Loaded answers from: ", report.computed_answers_file, "\n")
table_data = []
nL = 80
t_start = time.time()
score = {}
loader = SequentialTestLoader()
for n, (q, w) in enumerate(report.questions):
# q = q()
# q_hidden = False
# q_hidden = issubclass(q.__class__, Hidden)
if question is not None and n+1 != question:
continue
suite = loader.loadTestsFromTestCase(q)
......@@ -143,12 +133,11 @@ def evaluate_report(report, question=None, qitem=None, passall=False, verbose=Fa
q.possible = 0
q.obtained = 0
q_ = {} # Gather score in this class.
from unitgrade2.unitgrade2 import UTextTestRunner
# unittest.Te
# q_with_outstanding_init = [item.question for item in q.items if not item.question.has_called_init_]
from src.unitgrade2.unitgrade2 import UTextTestRunner
UTextResult.q_title_print = q_title_print # Hacky
UTextResult.show_progress_bar = show_progress_bar # Hacky.
UTextResult.number = n
UTextResult.nL = report.nL
res = UTextTestRunner(verbosity=2, resultclass=UTextResult).run(suite)
......@@ -157,20 +146,16 @@ def evaluate_report(report, question=None, qitem=None, passall=False, verbose=Fa
assert len(res.successes) + len(res.errors) + len(res.failures) == res.testsRun
# possible = int(ws @ possible)
# obtained = int(ws @ obtained)
# obtained = int(myround(int((w * obtained) / possible ))) if possible > 0 else 0
obtained = int(w * obtained * 1.0 / possible ) if possible > 0 else 0
score[n] = {'w': w, 'possible': w, 'obtained': obtained, 'items': q_, 'title': qtitle}
q.obtained = obtained
q.possible = possible
s1 = f"*** Question q{n+1}"
s1 = f" * q{n+1}) Total"
s2 = f" {q.obtained}/{w}"
print(s1 + ("."* (nL-len(s1)-len(s2) )) + s2 )
print(s1 + ("."* (report.nL-len(s1)-len(s2) )) + s2 )
print(" ")
table_data.append([f"Question q{n+1}", f"{q.obtained}/{w}"])
table_data.append([f"q{n+1}) Total", f"{q.obtained}/{w}"])
ws, possible, obtained = upack(score)
possible = int( msum(possible) )
......@@ -185,10 +170,12 @@ def evaluate_report(report, question=None, qitem=None, passall=False, verbose=Fa
seconds = dt - minutes*60
plrl = lambda i, s: str(i) + " " + s + ("s" if i != 1 else "")
print(f"Completed: "+ dt_string + " (" + plrl(minutes, "minute") + ", "+ plrl(seconds, "second") +")")
from src.unitgrade2.unitgrade2 import dprint
dprint(first = "Total points at "+ dt_string + " (" + plrl(minutes, "minute") + ", "+ plrl(seconds, "second") +")",
last=""+str(report.obtained)+"/"+str(report.possible), nL = report.nL)
# print(f"Completed at "+ dt_string + " (" + plrl(minutes, "minute") + ", "+ plrl(seconds, "second") +"). Total")
table_data.append(["Total", ""+str(report.obtained)+"/"+str(report.possible) ])
results = {'total': (obtained, possible), 'details': score}
return results, table_data
__version__ = "0.0.2"
\ No newline at end of file
Metadata-Version: 1.0
Name: unitgrade
Version: 0.0.5
Summary: A lightweight student evaluation framework build on unittest
Home-page: https://lab.compute.dtu.dk/tuhe/unitgrade
Author: Tue Herlau
Author-email: tuhe@dtu.dk
License: Apache
Description: UNKNOWN
Platform: UNKNOWN
MANIFEST.in
README.md
setup.py
cs101courseware_example/Report0_resources_do_not_hand_in.dat
cs101courseware_example/Report1_resources_do_not_hand_in.dat
cs101courseware_example/Report2_resources_do_not_hand_in.dat
cs101courseware_example/__init__.py
cs101courseware_example/cs101report1.py
cs101courseware_example/cs101report1_grade.py
cs101courseware_example/cs101report2.py
cs101courseware_example/cs101report2_grade.py
cs101courseware_example/homework1.py
cs101courseware_example/instructions.py
unitgrade/__init__.py
unitgrade/unitgrade.py
unitgrade/unitgrade_helpers.py
unitgrade.egg-info/PKG-INFO
unitgrade.egg-info/SOURCES.txt
unitgrade.egg-info/dependency_links.txt
unitgrade.egg-info/requires.txt
unitgrade.egg-info/top_level.txt
\ No newline at end of file
jinja2
tabulate
sklearn
compress_pickle
cs101courseware_example
unitgrade
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment