Skip to content
Snippets Groups Projects
Select Git revision
  • 1f65e4cdcf21ad1cba8be17ac7d657dd7e2cfd8d
  • master default protected
  • develop
  • release-2019.0
  • feature/slupdate
  • releases/2018
  • devel
  • feature/folder-reorg
  • releases/2017
  • releases/2016
10 results

dtu-enote

  • Clone with SSH
  • Clone with HTTPS
  • dtu-enote

    This is the setup we want to end up with.

    http:letsencrypt-enote.compute.dtu.dk    used only to renew certs
    
    https://enote.compute.dtu.dk             a landing site with just a static html file
    
    https://quiz.compute.dtu.dk              shared quiz for all courses
    https://file-admin.compute.dtu.dk        shared File Admin for all courses
    https://stackedit.compute.dtu.dk         shared Stack Edit for all courses
    
    https://01005.compute.dtu.dk/            course website (rails)
    https://01005.compute.dtu.dk/askbot      askbot just for 01005
    https://01005.compute.dtu.dk/sharelatex  sharelatex just for 01005

    To simulate that setup I would like to end up with

    http:letsencrypt-enote.compute.dtu.dk       used only to renew certs
    
    https://testlanding.compute.dtu.dk          a landing site with just a static html file
    
    https://testsite1.compute.dtu.dk/           course website (rails)
    https://testsite1.compute.dtu.dk/askbot     askbot site just for "testsite1".

    where each domain is its own container.

    ./run.sh sets that up, but where https is missing.

    The idea is that letsencrypt-enote shares an volume, containing the SSL certs, with the reverse proxy container, so the certs can be updated and in worst case also have to reload the reverse proxy container.

    Maybe quiz wouldn't be in the reverse proxy, as it might have deep letsencrypt integrations. Ask Iain.

    Organization and Setup

    This is the root project for enote. It consists of a reverse proxy setup and a number of child projects (in separate repos) to which the main server proxies requests. Each project (and the nginx proxy frontend) are run in separate containers.

    dtu-enote/components dtu-enote/components/askbot dtu-enote/components/stackedit dtu-enote/components/file-manager dtu-enote/components/course-website dtu-enote/components/couchdb dtu-enote/components/dtu-quiz

    course-website    -> dtu-enote-website
    docker-askbot     -> dtu-enote-askbot
    docker-cn-dumper  -> dtu-enote-cndumper
    docker-couchdb    -> dtu-enote-couchdb
    docker-get-pdf    -> dtu-enote-getpdf
    docker-mongodb    -> dtu-enote-mongodb
    docker-sharelatex -> dtu-enote-sharelatex
    stackedit         -> dtu-enote-stackedit
    dtu-core          -> dtu-enote-core
    quiz-parser       -> dtu-quiz-parser
    pdf.js            -> dtu-enote-pdfjs
    
    
    repos=("dtu-enote-website"
    "dtu-enote-askbot"
    "dtu-enote-cndumper"
    "dtu-enote-couchdb"
    "dtu-enote-getpdf"
    "dtu-enote-mongodb"
    "dtu-enote-sharelatex"
    "dtu-enote-stackedit"
    "dtu-enote-core"
    "dtu-quiz-parser"
    "dtu-enote-pdfjs"
    "dtu-enote"
    "dtu-quiz"
    "dtu-enote-core" )
    
    for r in ${repos[@]}
    do
      echo $r
      git clone -b master https://github.com/dtu-compute/$r
      pushd $r
      git fetch origin
      popd
    done
    
    for r in ${repos[@]}
    do
      echo $r
      pushd $r
      git fetch origin 
      git log --left-right --graph --cherry-pick --oneline origin/2016...origin/devel
      popd
    done
    
    
    pushd components
    branch=$(git symbolic-ref --short HEAD)
    git clone -b $branch https://github.com/dtu-compute/dtu-enote-sharelatex sharelatex
    git clone -b $branch https://github.com/dtu-compute/dtu-enote-installer
    git clone -b $branch https://github.com/dtu-compute/dtu-enote-cndumper cndumper
    git clone -b $branch https://github.com/dtu-compute/dtu-enote-couchdb couchdb
    git clone -b $branch https://github.com/dtu-compute/dtu-enote-website course-website
    git clone -b $branch https://github.com/dtu-compute/dtu-enote-mongodb mongodb
    git clone -b $branch https://github.com/dtu-compute/dtu-enote-core dtu-core
    git clone -b $branch https://github.com/dtu-compute/dtu-quiz-nginx dtu-quiz-nginx
    git clone -b $branch https://github.com/dtu-compute/dtu-enote-landing landing
    git clone -b $branch https://github.com/dtu-compute/dtu-enote-letsencrypt letsencrypt
    git clone -b $branch https://github.com/dtu-compute/dtu-enote-course-website
    git clone -b $branch https://github.com/dtu-compute/dtu-enote-stackedit
    git clone -b $branch https://github.com/dtu-compute/dtu-enote-docker-couchdb couchdb
    git clone -b $branch https://github.com/dtu-compute/dtu-enote-docker-getpdf get-pdf
    
    find . -name .git -type d -prune 
    find components -name .git -type d -prune -exec git --git-dir={}/ --work-tree=$PWD/{}/.. pull origin devel \;
    
    find components -name .git -type d -prune -exec git --git-dir={}/ --work-tree=$PWD/{}/.. status \;
    
    
    for i in components/*/.git; do ( echo $i; cd $i/..; git pull; ); done
    
    popd

    Building

    alias dep='bundle exec rake deploy:all'
    ./run.sh -disable:sharelatex -disable:askbots -disable:getpdf

    shared folders

    chcon -Rt svirt_sandbox_file_t /filemanager/

    echo "

    hi" > /foo/index.html

    Debugging

    Shell Into Containsers

    docker exec -i -t landing /bin/bash 
    docker exec -i -t letsencrypt /bin/bash 
    docker exec -i -t proxy /bin/bash 
    docker exec -i -t couchdb /bin/bash 
    docker exec -i -t stackedit /bin/bash 
    docker exec -i -t mongodb /bin/bash 
    docker exec -i -t website-00000 /bin/bash
    docker exec -i -t website-01005 /bin/bash
    docker exec -i -t website-01006 /bin/bash
    docker exec -i -t website-02402 /bin/bash
    docker exec -i -t file-manager /bin/bash
    docker exec -i -t get-pdf /bin/bash
    docker exec -i -t askbot-00000 /bin/bash
    docker exec -i -t askbot-01005 /bin/bash
    docker exec -i -t quiz /bin/bash
    docker exec -i -t quiz-db /bin/bash
    docker exec -i -t mongodb-sharelatex /bin/bash
    docker exec -i -t sharelatex-01005 /bin/bash
    
    
    ## View Logs
    
    ### Container Logs
    

    docker logs file-manager docker logs quiz_1

    
    
    ### Logs from Applications

    docker exec -i -t course-website-00000 cat /dtu-course/log/production.log docker exec -i -t website-01005 cat /dtu-course/log/production.log docker exec -i -t website-02402 cat /dtu-course/log/production.log docker exec -i -t website-02323 cat /dtu-course/log/production.log docker exec -i -t file-manager cat /file-manager/app/log/production.log docker exec -i -t quiz cat /dtu-quiz/app/log/production.log | tail -500 docker exec -i -t quiz cat /dtu-quiz/app/log/populate.log docker exec -i -t quiz cat /file-manager/app/log/production.log docker exec -i -t askbot-01005 cat /data/log/askbot.log

    docker exec -i -t letsencrypt cat /var/log/nginx/error.log

    docker exec -i -t letsencrypt cat/var/log/nginx/access.log

    docker exec -i -t sharelatex-00000 cat /var/log/sharelatex/web.log

    
    
    ## Populate redis with identity information
    

    docker exec -i -t quiz bundle exec rake populate docker exec -i -t dtuenotedevel_quiz bundle exec rake populate

    
    
    ## DB Operations
    
    ### Clean couchdb
    

    docker exec -i -t couchdb /clean-db.sh rm -rf /usr/local/var/lib/couchdb/*

    
    ### Shutdown DBs
    

    docker exec -i -t couchdb /shutdown.sh docker exec -i -t mongodb /shutdown.sh docker exec -ti sharelatex-db killall mongod

    
    
    more /var/log/nginx/access.log
    tail /var/log/nginx/access.log
    
    # find rails failures
    docker logs course_website | grep -A 5 -B 5 "\(ERROR\|FATAL\)"

    CRON

    cp crontab /etc/cron.d/populate-cron chmod 0644 /etc/cron.d/populate-cron touch /enote/vol/log/quiz/populate.log crontab -e

    Enabling and disabling components

    ./run.sh disable:sharelatex disable:letsencrypt disable:askbot

    Adding new Courses

    1. Update the courses. Add the new one with enabled: false in courses.yaml
    /enote/vol/config/courses.yaml
    /enote/vol/config/cn-dumper.yaml
    1. (Optional) run.sh

    2. Create new certs

    ./make-certs.sh .compute.dtu.dk

    Note: the URL suffix is required

    1. Update the courses.yaml to have enabled:true

    2. Restart

    run.sh

    Re-uploading raw Web site files to couchdb

    1. Make sure node is installed
    curl --silent --location https://rpm.nodesource.com/setup_6.x | bash -
    yum install -y nodejs
    1. Set up NPM
    pushd components/stackedit
    npm install
    1. Upload files
     ./couchdb_import.sh 01006 /enote/vol/website-raw/01006
     ./couchdb_import.sh 00000 /enote/vol/website-raw/00000
     ./couchdb_import.sh 02402 /enote/vol/website-raw/02402

    Updating course Web site

    docker exec -i -t website-00000 bundle exec rake update:all
    docker exec -i -t website-00000 bundle exec rake update:compile_pages
    docker exec -i -t website-00000 bundle exec rake update:podcasts
    docker exec -i -t website-02323 bundle exec rake update:all
    docker exec -i -t quiz bundle exec rake dtuquiz:dump_course['00000']

    Errors

    This can happen:

    There were too many requests of a given type :: Error creating new cert :: Too many certificates already issued for: dtu.dk
    Please see the logfiles in /var/log/letsencrypt for more details.
    askbot-00000-enote.compute.dtu.dk

    Helpful commands to investigate:

    docker exec -i -t letsencrypt ls /var/log/letsencrypt
    docker exec -i -t letsencrypt /bin/bash 

    Building and Pushing Sharelatex Containers

    We use two docker hub containers: one for the base sharelatex and one for the enote-specific sharelatex.

    sharelatex-enote-base

    We build our own version of the base sharelatex image to ensure that it doesn't change (and, initially at least, to fix the tlmgr bug ). The sharelatex image is itself based on a base image with another layered on top. To build it it the enote project root:

    pushd components
    git clone https://github.com/sharelatex/sharelatex-docker-image.git
    popd
    ./build-sharelatex-enote-base.sh

    This will concatenate the two Dockerfiles describing the sharelatex image, build it, and publish it. docker login may be required in order to push.

    sharelatex-enote

    After the base is build, you can build the enote-specific image:

    pushd components/sharelatex
    ./build-sharelatex-enote.sh

    enote itself will simply use the sharelatex-enote image.