Introduction

Stanford GSB maintains research computing servers for the almost-exclusive use of GSB faculty, staff, and students. Our pool of computers, which we call the yens, are an interactive computing environment with over 128 cores, 6 TB of RAM, 6-8 TB local storage and about 500 TB networked Isilon storage. About 100 users are active on the platform during a given academic year.

We say “almost-exclusive use” because GSB faculty can sponsor access for collaborating students, staff, or faculty from other departments or institutions. Two particularly common cases are inter-departmentally advised students (non-GSB students with a PhD advisor or co-advisor in GSB) and graduated GSB PhD students with faculty positions at other schools that are still collaborating with GSB faculty on projects.

At the beginning of AY 2018-2019 DARC (then RSS) launched a new automated process for granting, renewing, and auditing access to the yens. We created a new system primarily because minimum security guidelines required that we maintain quarterly access audits. This motivated us to change the technical means through which we granted access, which opened the door to additional changes. Our previous “process” for granting access was entirely human: potential collaborators would email us asking for access, we would email the sponsoring GSB faculty for approval, which we would provide if the faculty approved. Any record keeping or auditing we would have to create and manage ourselves on top of authorizing access.

DARC saw this as a perfect opportunity for automation through full-stack JS, technologies we had gained some proficiency with while supporting online experiments. Our new access sponsorship process, now in successful operation for almost a full academic year, has a single-page-app (SPA) through which collaborators can submit their own requests for sponsored access and faculty can view and approve or reject requests. A backend manages request storage and workgroup membership for yen access automatically. Access requests are valid for a quarter only, thus forcing a quarterly audit through a review and renew page available to sponsoring faculty. Using this tool DARC has processed about 100 sponsored access requests and/or renewals largely without any intervention.

Our App

Let’s first review our react.js app. This is a single page app (SPA) allowing

  • requesters to submit requests for access,
  • faculty to view and approve requests,
  • and faculty to renew requests near quarter turnovers

all without our involvement. Other features enable DARC access for reviewing request status and/or approving/rejecting requests on behalf of faculty for when our involvement is needed.

Function control is managed by specifying standard URL “query” parameters:

  • / is for submitting requests
  • /?approve=... is for approving requests
  • /?renew=... is for renewing requests
  • /?review=... is for reviewing requests (DARC only)

Any page load is SSO protected, using our rssreacts template which we won’t discuss here.

Requests

A SUNetID is required for identification on the yens, and anyone with a valid SUNet ID can submit a request for sponsored access. The request form is hosted at

https://yen.gsbrss.com/non-gsb-access/

This request page is a form like:

Some parameters in the form – the requester’s name and SUNetID – are autofilled using login data. The sponsoring faculty SUNetID/name fields are also connected and autocomplete from a list of identities drawn from the current GSB faculty workgroup.

Upon successful submission of a request, two things happen concurrently: DARC is notified of the request via a Slack message like

and the sponsoring faculty also recieves a formatted email with a custom URL to approve the request

This image is from our development environment, and thus has a different web address. Notice though that this URL has the parameter approve=xxxx for some hex-encoded token identifying the request to approve. For security purposes, only the sponsoring faculty listed in a request (or a member of DARC) can see request-specific approval pages. This is managed by the SSO process: if someone other than the listed sponsoring faculty (or a member of DARC) logs onto the approval page for a request, they get a “you cannot approve this request” notice and nothing else.

Approvals

When a sponsoring faculty clicks on the link sent in the email (and logs in) they see an approval page of the form:

Here they are asked to verify the requesters SUNetID and approve or reject the request. We feel asking faculty to verify knowledge of the requester’s SUNetID is a reasonably minimal verification step to prove the two parties are at least in contact with each other about access.

If the sponsoring faculty approves the request, they see (after the approval processes) the following success notification in their browser:

DARC also recieves a notification of the approval via Slack:

The requester also recieves an email notifying them that their request has been approved:

More importantly, the app backend automatically adds the requester to the appropriate quarter’s access workgroup through Stanford’s MaIS API. Our database is also updated with this step.

Note also that a request can’t be approved twice: If the faculty revisits the same link, they get a message to this effect:

Renewals

The renewal page is triggered by including a renew query parameter in the URL. However, renewals are only active “near” quarter turnovers. If the page is accessed outside of the appropriate window, renewals are blocked:

Sponsoring faculty recieve a renewal notice email near the end of each quarter and after each new quarter begins.

Review

The review page, accessible only to DARC team members looks something like this:

We use this page to succinctly monitor the state of access requests in a given quarter, as well as to check up on specific requests when needed.

Code Structure

The best way to learn about the app code is to clone and review the repo. We’ll provide an sketch overview here.

As mentioned above, we built our app on top of rssreacts, a template react.js project for integrating with Stanford identity elements and SSO. We use redux.js for state management and build the UI off of MaterialUI components (but, sadly, using version 0).

Our js/jsx source is organized as follows:

    src
    |__ actions
    |__ components
    |__ reducers
    |__ Stanford
    |__ constants.js
    |__ index.js
    |__ store.js
    |__ styles.js

actions, components, reducers, and Stanford are folders that contain, respectively, redux actions, react components, redux state reducers, and finally Stanford identity customizations of MaterialUI components.

The main customized section here is components, which contains the following:

    src
    |__ components
        |__ App.js
        |__ DesktopApp.js
        |__ TabletApp.js
        |__ MobileApp.js
        |__ RequestForm.js
        |__ ApproveForm.js
        |__ RenewForm.js
        |__ ReviewForm.js
        |__ Header.js
        |__ NoteField.js
        |__ note.css

App.js pretty much only filters for the display size and renders a specific component: DesktopApp, TabletApp, or MobileApp. Currently, only the DesktopApp is fully defined; TabletApp and MobileApp require further checking that the UI is appropriately visible and functional. DesktopApp filters based on login data and query parameters and renders various error messages and/or specific forms, and also has an ErrorBoundary to catch unforseen UI errors and submit information about them to our backend. Each of our “forms” is defined as its own component in the corresponding file: RequestForm.js, ApproveForm.js, RenewForm.js, and ReviewForm.js. For the most part, each of these forms are relatively easy to understand, if highly application specific, react components.

Header.js contains a generic user-customized header for any visible page. NoteField.js is a draft.js flexible text field, and note.css contains some css used therein.

Backend Server

DARC composed a node.js backend to serve mainly as an interface to a MongoDB Atlas database, our permanent store of access requests and their status. Two other functions of our backend are to interact with our actual access API upon approval as well as run a daily “check” for several key functions:

  • database connection verification,
  • request reminders,
  • request expiration,
  • and renewal notices near quarter changes.

Overall the backend is a fairly simple, essentially single-file server. In this section we review its functions and how it works.

Routes

The backend creates an express server that responds to the following routes with the associated summarized functions:

  • GET /: A basic “hello world” response from the server
  • GET /quarter: get details for the current quarter from the server
  • GET /quarters: get all quarters from the server
  • GET /approvers: get all “approvers” from the server
  • HEAD|GET /isapprover/:uid: Check whether a user (uid) is an approver
  • HEAD|GET /isadmin/:uid: Check whether a user (uid) is an admin
  • GET /requests: Get requests for yen access for the current quarter
  • GET /requests/:quarter: Get requests for yen access for a specific quarter, quarter, specified as season-year
  • PUT|POST /request: declare a new request for sponsored access to the yens
  • GET|POST /approve/:id/:token: return request information if appropriate for GET calls, and process approval if appropriate for POST calls
  • POST /reject/:id/:token: reject a request for access, if appropriate
  • GET /renewals/:uid: get list of renewals that could be processed for a given user (if they are an approver)
  • POST /renew/:token: renew requests specified in the request body, if appropriate, for the next quarter
  • PUT|POST /error: declare a new front-end error to store in the database

Logging

We log all requests and their responses as follows:

[2019-04-29T17:51:35.730] [INFO] info - REQLOG,GET,/QUARTER,a0152459d625082746722256,VoQt8NUNupoz73IQb08...
[2019-04-29T17:51:35.731] [INFO] info - RESLOG,a0152459d625082746722256,GET,/QUARTER,200,JSON,1556560295.730,1556560295.731,0.001
[2019-04-29T17:51:35.731] [INFO] info - REQLOG,GET,/APPROVERS,9d618f069246d95f9e0fa5c4,VoQt8NUNupoz73IQb08...
[2019-04-29T17:51:35.732] [INFO] info - RESLOG,9d618f069246d95f9e0fa5c4,GET,/APPROVERS,200,JSON,1556560295.731,1556560295.732,0.001
[2019-04-29T17:52:39.794] [INFO] info - REQLOG,POST,/REQUEST,5596692be7fbc64394527579,VoQt8NUNupoz73IQb08...
[2019-04-29T17:52:42.570] [INFO] info - RESLOG,5596692be7fbc64394527579,POST,/REQUEST,200,JSON,1556560359.794,1556560362.570,2.776

“Request” and “Response” log lines are started with REQLOG and RESLOG, respectively. The content in the REQLOG line format is as follows:

<method>,<URL>,<response key>,<login token>

The content in the RESLOG line format is

<response key>,<method>,<URL>,<status>,<response method>,<start time>,<end time>,<duration>

Note that the <response key> field is a randomly generated, presumably unique field that can be used to join request/response data if desired. However, the different entries are reasonably well partitioned already. If you wanted to know how long requests were taking to process, for example, you only need to examine the RESLOG lines:

$ sed -En '/RESLOG/p' ${LOG} | awk -F',' '{ print $9 }'
GET /APPROVERS,0.000
GET /QUARTER,0.000
GET /APPROVERS,0.001
GET /ISADMIN/MORROWWR,0.001
GET /REQUESTS,0.325
GET /QUARTERS,0.001
GET /ISADMIN/MORROWWR,0.000
GET /REQUESTS,0.165
...

or

$ sed -En '/RESLOG(.*)\/REQUESTS/p' ${LOG} | awk -F',' '{ print $3" "$4","$9 }'
GET /REQUESTS,0.159
GET /REQUESTS,0.247
GET /REQUESTS,0.153
GET /REQUESTS/SPRING%202019,0.083
GET /REQUESTS/WINTER%202019,0.154
GET /REQUESTS,0.245
GET /REQUESTS,0.227
GET /REQUESTS,0.159
...

Daily Check Loop

Our server runs a set sequence of tasks every 24 hours. This “check loop” is run using our coordinator class for asynchronous processing on directed acyclic graphs.

  • refresh approvers: get configured list of workgroups that are “approvers” and create an internal list
  • refresh admins: get configured list of workgroups that are “admins” and create an internal list
  • reload MongoDB: refresh the connection to the MongoDB Atlas service
  • reload quarters (after MongoDB): get the list of quarters from MongoDB and process it into a linked list form
  • define "this" quarter (after quarters): define the local variable for the quarter that corresponds to the actual date
  • email DARC (after MongoDB): send a simple notification to DARC staff that the check loop has “fired”
  • deduplicate pending requests: look through the database for repeated pending requests, and delete all but one
  • remind approvers and expire requests: remind approvers that they have pending requests and/or expire requests that have been pending too long
  • check renewal notice (after defining quarter): check if we should send out a renewal notice to all faculty with sponsorships in the current quarter

Workgroup Integration

Routines and how they work

Notifications

Emails (pug templates, AWS SES), Slack

We have created several email templates we use in different situations:

  • notifysponsor.pug: notify a sponsor that an access request listing them has been submitted
  • remindsponsor.pug: sent to the sponsor before an access request will expire
  • approved.pug: sent to requester when their access request was approved
  • rejected.pug: sent to requester when their access request was rejected
  • expired.pug: sent to requester when their access request expires without action
  • renewals.pug: sent to a sponsor when it is time to renew sponsored access
  • renewed.pug: sent to requester when their access is renewed for another quarter

Configuration

We use config.json to enable use of json encoded configuration files. INI style config files are a bit easier to use, but json isn’t much worse and is pretty consistent with the JS code used here.

Configurable options are

  • port: the port to run the server on
  • logs: a dictionary with a specification of the logs to write out
  • workgroups: client SSL certs for the MaIS API
  • appURL: URL of the front-end, so emails can be appropriately constructed
  • approvers: list of workgroups to draw approvers from
  • admins: list of workgroups to draw admins from
  • runCheckLoopAt: UTC time to start running the check loop at
  • checkLoopInterval: A duration in between check loop runs, in ISO 8601 duration format
  • daysRequestValid: The number of days before a request for access expires
  • renewDaysFromEnd: The number of days before a quarter’s end to send a proactive renewal notice
  • renewDaysAfterStart: The number of days after a quarter starts to send a renewal reminder notice
  • login: login server (if using basic authentication), as a dictionary with fields protocol, host, port, and path
  • database: dictionary of MongoDB database connection information, each value as a dictionary with fields service, uri, name, formalName, and collections
  • use_database: the key of the database in databases to use
  • use_basic_auth: whether or not to use basic authorization
  • aws_config: AWS credentials with permissions for SES services

Here is an example configuration:

{
    "environment" : "development" , 
    "port"           : "5000" , 
    "logs" : {  
        "info"  : [ { "name" : "out" } , { "name" : "log" , "file" : "info.log" } ] , 
        "error" : [ { "name" : "err" , "file" : "error.log" } ] 
    } , 
    "workgroups" : {
        "cert"  : "/path/to/mais/workgroup/cert.cert" , 
        "key"   : "/path/to/mais/workgroup/key.key"
    } , 
    "appURL" : "https://yen.gsbrss.com:5000" , 
    "approvers" : [ "gsb_acl:faculty" ] , 
    "admins" : [ "gsb-rc:admin" ] , 
    "runCheckLoopAt"    : "08:00:00" , 
    "checkLoopInterval" : "P1DT0H0M0S" , 
    "daysRequestValid" : 7 ,
    "renewDaysFromEnd" : 28 , 
    "renewDaysAfterStart" : 14 , 
    "login" : {
        "protocol" : "https" , 
        "host" : "yen.gsbrss.com" , 
        "port" : "" , 
        "path" : "login"
    } , 
    "database" : {
        "prod" : {  
            "service"       : "mongodb" , 
            "uri"           : "<mongodb atlas connection string for prod database>" , 
            "name"          : "yenaccess-prod" , 
            "formalName"    : "Production Data" , 
            "collections"   : { "requests"  : "requests" , "quarters" : "quarters" , "errors" : "errors" } 
        } , 
        "test" : {  
            "service"       : "mongodb" , 
            "uri"           : "<mongodb atlas connection string for test database>" , 
            "name"          : "yenaccess-test" , 
            "formalName"    : "Test/Dev Data" , 
            "collections"   : { "requests"  : "requests" , "quarters" : "quarters" , "errors" : "errors" } 
        }
    } , 
    "use_database" : "test" , 
    "use_basic_auth" : "no" , 
    "aws_config" : {
        "accessKeyId"     : "AN0AWS0ACCESSKEY0ID0" ,
        "secretAccessKey" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" , 
        "region"          : "us-west-2"
    } 
}

Database

As mentioned above, we have a MongoDB database in the Atlas service to store request data. We also store data about academic quarters as well as UI errors from a react.js ErrorBoundary. Accordingly, our (testing and production) databases have three collections for requests, quarters, and errors. We discuss the content of each below.

Requests

A request json object looks like something of the form

{ 
    "_id" : ObjectId("xxxxxxxxxxxxxxxxxxxxxxxx"), 
    "submitter" : { "name" : "Christina Zhu", "sunetid" : "xxxxxx", "address" : "XXX.XXX.XXX.XXX" }, 
    "faculty" : { "name" : "David Larcker", "sunetid" : "xxxxxx" }, 
    "user" : { "name" : "Christina Zhu", "sunetid" : "xxxxxx", "dept" : "...", "rank" : "...", "contact" : "..." }, 
    "request" : {
        "time" : "2019-04-15T19:14:35.805Z", 
        "quarter" : "spring 2019", 
        "description" : "Compensation peers project\n", 
        "ifsAccess" : [ "/ifs/gsb/czhu" ], 
        "other" : "\n"
    }, 
    "status" : "approved", 
    "history" : [
        { "event" : "request-received", "time" : "2019-04-15T19:14:35.998Z" }, 
        { "event" : "email-notified-sponsor", "time" : "2019-04-15T19:14:39.768Z" }, 
        { "event" : "request-viewed", "user" : "xxxxxx", "time" : "2019-04-15T19:56:04.695Z" }, 
        { "event" : "request-viewed", "user" : "xxxxxx", "time" : "2019-04-15T19:58:14.382Z" }, 
        { "event" : "request-viewed", "user" : "xxxxxx", "time" : "2019-04-15T20:02:40.856Z" }, 
        { "event" : "request-approved", "user" : "xxxxxx", "time" : "2019-04-15T20:02:54.821Z" }, 
        { "event" : "email-request-approved", "time" : "2019-04-15T20:02:55.040Z" }
    ]
}

Here we use xs to redact information that might be sensitive and elipses ... for extraneous details.

Quarters

Quarter json objects start as something like

{ 
    "_id" : ObjectId("xxxxxxxxxxxxxxxxxxxxxxxx"), 
    "season" : "spring", "year" : "2018", "start" : "2018-04-01", "end" : "2018-06-01", 
    "next" : { "season" : "summer", "year" : "2018" }
}
{ 
    "_id" : ObjectId("xxxxxxxxxxxxxxxxxxxxxxxx"), 
    "season" : "summer", "year" : "2018", "start" : "2018-06-15", "end" : "2018-08-15", 
    "next" : { "season" : "fall", "year" : "2018" }
}

This initial data stores a quarter’s “season” and year, the starting and ending days for that quarter, and a quarter’s successor in the next field. The next specifications allow us to easily construct a proper linked list for quarters in the backend.

When we send reminders about quarter transitions, additional information is stored in the quarter objects:

{ 
    "_id" : ObjectId("xxxxxxxxxxxxxxxxxxxxxxxx"), 
    "season" : "fall", "year" : "2018", "start" : "2018-09-14", "end" : "2018-12-07", 
    "next" : { "season" : "winter", "year" : "2019" }, 
    "renewals" : { "notice" : true, "reminder" : true }, 
    "history" : [
        { "event" : "sent-reminder", "user" : "xxxxxx", "time" : "2019-01-11T17:47:17.007Z" }, ...
    ]
}

The renewals field stores flags for whether a notice email has been sent (as the current quarter is ending) and whether a reminder email has been sent (after a new quarter starts). Each notice and reminder sent is also logged in the history field to help avoid sending repeated notifications to faculty.

Errors

Error json objects look something like:

{ 
    "_id" : ObjectId("xxxxxxxxxxxxxxxxxxxxxxxx"), 
    "time" : "2018-12-16T15:47:57.529Z", 
    "message" : "TypeError: Cannot read property 'map' of undefined", 
    "error" : { }, 
    "stack" : [
        "", 
        "in ReviewForm (created by Connect(ReviewForm))", 
        "in Connect(ReviewForm) (at DesktopApp.js:297)", 
        ...
    ], 
    "request" : {
        "host" : "localhost:5000", 
        "connection" : "keep-alive", 
        "content-length" : "865", 
        "accept" : "application/json, text/plain, */*", 
        "accepts" : "application/json", 
        "origin" : "http://localhost:3000", 
        "authorization" : "0", 
        "user-agent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 ... ", 
        "content-type" : "application/json", 
        "referer" : "http://localhost:3000/build/auth?login=0&review=1", 
        "accept-encoding" : "gzip, deflate, br", 
        "accept-language" : "en-US,en;q=0.9"
    }
}

This data describe an error in terms of it’s occurrence time, message, call stack trace, and request data from the client that posted the error (and thus the client that observed the error). In this case, the error was logged by a server running locally (localhost:5000) sent from a client also running locally (localhost:3000) during debugging.

Usage

Conclusions