NGL REST API Documentation

This page describes the read-only REST API for querying data on Next Generation Liquefaction: authentication, available endpoints, and the query-string syntax used to filter, sort, select, and paginate results. The REST API provides an alternative to browsing and downloading data using the website. You can query data directly from any software package that supports HTTP requests, which is well-suited to developing end-to-end workflows. We provide examples below using Python and its "requests" package.

Authentication

Every endpoint requires a bearer token, so that only authenticated users can pull data and so we can tell which data a given user is allowed to see. Request a token by sending your account's username or email and password as HTTP Basic Auth to: 

GET https://nextgenerationliquefaction.org/users/api-token

GET is shown above since this endpoint doesn't need a request body anyway (your credentials go in the Basic Auth header, not the body), but POST works here too - see Request Method below. A token is valid for 2 hours from the time it's issued. After that, request a new one the same way - we suggest never hard-coding your username and password directly into a script; prompt for them instead, as below. 

import requests
from requests.auth import HTTPBasicAuth
import getpass
import json

email = input('email: ')
password = getpass.getpass('password: ')
url = 'https://nextgenerationliquefaction.org/users/api-token'
headers = {}
headers['User-Agent'] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) 134.0.6998.118 Safari/537.36"
headers['Accept'] = "application/json"
r = requests.get(url, headers=headers, auth=HTTPBasicAuth(email, password))
token = json.loads(r.text)['token']

Include that token on every subsequent request as a bearer token:

headers = {}
headers['Accept'] = "application/json"
headers['User-Agent'] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) 134.0.6998.118 Safari/537.36"
headers['Authorization'] = "Bearer {}".format(token)
r = requests.get('https://nextgenerationliquefaction.org/sites/api-index', headers=headers)

A missing, malformed, expired, or incorrectly-signed token returns HTTP 400 with an errorMessage explaining why.

Request Method

Every endpoint accepts GET, with all parameters - filtering, sorting, selecting fields, and pagination - passed as URL query-string parameters. This is the simplest option and works for most requests.

Every endpoint also accepts POST, with the same parameters sent as the request body (form-encoded or JSON) instead of the query string - useful when a where= or select= value is too long to comfortably fit in a URL. If a parameter is given both ways on the same request, the POST body value wins.

Here is an example request with some query string parameters, done as GET:

headers = {}
headers["Accept"] = "application/json"
headers["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) 134.0.6998.118 Safari/537.36"
headers["Authorization"] = "Bearer {}".format(token)
r = requests.get(
    'https://nextgenerationliquefaction.org/sites/api-index?where=SITE_LAT>30 AND SITE_LAT<40&select=SITE_ID,SITE_NAME,SITE_LAT,SITE_LON',
    headers=headers,
)

...and as POST, with those same parameters moved into the request body via data= instead of as query string parameters - the URL itself stays just the base endpoint, which is the point when select=/where= get long (e.g. dozens of field names): 

headers = {}
headers["Accept"] = "application/json"
headers["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) 134.0.6998.118 Safari/537.36"
headers["Authorization"] = "Bearer {}".format(token)
r = requests.post(
    'https://nextgenerationliquefaction.org/sites/api-index',
    headers=headers,
    data={
        'select': 'SITE_ID,SITE_NAME,SITE_LAT,SITE_LON',
        'where': 'SITE_LAT>30 AND SITE_LAT<40',
    },
)

Both return the identical result.

URL Structure

A request has three parts:


        https://nextgenerationliquefaction.org/sites/api-index?limit=20&sort=SITE_NAME&direction=asc
        \______________base_url______________/\___endpoint____/\_________query_string_______________/
  • base_url - https://nextgenerationliquefaction.org
  • endpoint - one resource's path, e.g. /sites/api-index (see the full list below)
  • query_string - ? followed by &-separated key=value pairs

Available Endpoints

Each endpoint corresponds to one database table and returns a plain JSON array of row objects (not wrapped in a named key) - [{...}, {...}, ...]. The "Review status" column is explained in the What Data is Returned (Review Status) section. The tables below cover the app's main resources; the join between two of them is exposed separately, in Junction (many-to-many) endpoints.

Data Endpoint Table Review status
Sites/sites/api-indexSITEStandard
Field tests/field-tests/api-indexTESTStandard
Field observations/field-observations/api-indexFLDOStandard
Samples/samples/api-indexSAMPStandard
Specimens/specimens/api-indexSPECStandard
Plasticity tests/plasticity-tests/api-indexPLASStandard
Other tests/other-tests/api-indexOTHRStandard
Index tests/index-tests/api-indexINDXStandard
Density measurements/density-measurements/api-indexRDENStandard
Consolidation tests/consolidation-tests/api-indexCONGStandard
Consolidation test stages/consolidation-test-stages/api-indexCON_STGEStandard
Triaxial tests/triaxial-tests/api-indexTXGStandard
Triaxial test stages/triaxial-test-stages/api-indexTXSStandard
Direct simple shear tests/direct-simple-shear-tests/api-indexDSSGStandard
Direct simple shear test stages/direct-simple-shear-test-stages/api-indexDSSSStandard
Particle size distributions/grag/api-indexGRAGStandard
Particle size distribution data/grat/api-indexGRATStandard
Penetration test results (SPT blow counts)/penetration-test-results/api-indexISPTStandard
Boreholes/boreholes/api-indexBORHStandard
Depth/layer descriptions/depth-layer-descriptions/api-indexDETLStandard
Stratigraphic layer descriptions/stratigraphic-layer-descriptions/api-indexSTRAStandard
Test pits/test-pits/api-indexTEPTStandard
Ground water tests/ground-water-tests/api-indexWATRStandard
Other field tests/other-field-tests/api-indexOTHFStandard
General cone penetration tests/general-cone-penetration-tests/api-indexSCPGStandard
Cone penetration data/cone-penetration-data/api-indexSCPTStandard
General invasive tests/general-invasives/api-indexGINVStandard
Invasive test details/invasive-details/api-indexGINDStandard
General surface wave investigations/general-surface-wave-investigations/api-indexGSWGStandard
Surface wave tests/surface-wave-tests/api-indexSWVGStandard
Surface wave test details/surface-wave-details/api-indexSWVDStandard
Surface wave dispersion curves/surface-wave-dispersion-curves/api-indexGSWDStandard
Displacement vectors/displacement-vectors/api-indexFLDDStandard
Liquefaction manifestations/liquefaction-manifestations/api-indexFLDMStandard
Ground motion intensity measurements/ground-motion-intensity-measurements/api-indexGMIMStandard
Files (metadata only, not file contents)/files/api-indexFILESpecial - see below
Events/events/api-indexEVNTNone (always all rows)
Stations/stations/api-indexSTATNone (always all rows)
Segments/segments/api-indexSEGMNone (always all rows)
Intensity measures/intensity-measures/api-indexIMNone (always all rows)
Ground motion types/gmim-types/api-indexGMIM_TYPENone (always all rows)
Ground motion components/gmim-comps/api-indexGMIM_COMPNone (always all rows)
Citations/citations/api-indexCITATIONNone (always all rows)
Labs/labs/api-indexLABNone (always all rows)
Lab programs/lab-programs/api-indexLAB_PROGRAMNone (always all rows)

Junction (many-to-many) endpoints

These endpoints expose the join between two of the resources above directly, rather than a resource you'd normally query on its own. Most carry nothing but the two foreign keys; a few (the *-files ones) also carry the file's caption. Their "Review status" follows whichever side of the join actually has a review workflow - see below.

Joins Endpoint Table Review status
Sites ↔ Files/site-files/api-indexSITFStandard
Field tests ↔ Files/field-test-files/api-indexTESFStandard
Field observations ↔ Files/field-observation-files/api-indexFLDFStandard
Samples ↔ Files/sample-files/api-indexSAMFStandard
Sites ↔ Citations/site-citations/api-indexSITPStandard
Field tests ↔ Citations/test-citations/api-indexTESPStandard
Field observations ↔ Citations/observation-citations/api-indexFLDPStandard
Lab programs ↔ Citations/lab-program-citations/api-indexLAB_PROGRAMPNone (always all rows)
Lab programs ↔ Samples/lab-programs-samples/api-indexLAB_PROGRAM_SAMPStandard
Field tests ↔ Samples/test-samples/api-indexSAMP_TESTStandard

Query Parameters

ParameterDefaultDescription
limit50Maximum number of rows to return.
page1Which page of results to return, 1-indexed.
sortthe table's primary keyField name to sort by.
directionascasc or desc (case-insensitive).
selectall fieldsComma-separated list of field names to return.
where(none)Filter expression - see below.
includeUnreviewedfalseWhether to include submitted-but-not-yet-reviewed rows - see below. Accepted (and silently ignored) on endpoints with Review Status marked "None" above.
role(none)Set to admin to bypass review-status filtering entirely - see below. Only valid value is admin; omit it entirely for normal (non-admin) behavior. If you do not have admin privileges, an error message will be returned.

There's no separate pagination metadata (no total row count, no "next page" field) in the response - it's always a plain array. If you need every row, keep requesting increasing page values with a fixed limit until you get back fewer rows than you asked for.

Filtering with where=

where= takes one or more conditions, each in the form FIELD OPERATOR VALUE, joined with AND, OR, and parentheses for grouping. A comma is treated the same as AND. AND/OR must be uppercase.

Field typeValid operators
Numeric / date / datetime= != > >= < <= IN NOT IN
Textall of the above, plus LIKE NOT LIKE
Boolean= !=

Examples:

where=SITE_LAT>30
where=SITE_LAT>30 AND SITE_LAT<40
where=SITE_LAT>30,SITE_LON<-100
where=SITE_NAME=Alameda OR SITE_ID=2
where=(SITE_LAT>30 AND SITE_LON<-100) OR SITE_NAME LIKE *Alameda*
where=TEST_TYPE IN (BORH,SCPG,TEPT)

For LIKE/NOT LIKE, use * as a wildcard for any number of characters - it's converted to SQL's % automatically.

As a shorthand, any query parameter that isn't one of the reserved names above (limit, page, sort, direction, select, where, includeUnreviewed) is treated as a plain equality filter and folded into where automatically, so ?SITE_NAME=Alameda and ?where=SITE_NAME=Alameda do the same thing. This is convenient for a single simple filter, but for anything involving AND/OR/operators other than equality, use where= directly.

Floating-point fields are stored with the usual floating-point rounding behavior - avoid =/!= on them (e.g. a stored 30.1 may not compare equal to a literal 30.1 in the query string); use a small range with >/< instead.

What Data Is Returned (Review Status)

Every site's data goes through a review workflow before it's public: a contributor submits it, and two reviewers later mark it reviewed. The REST API respects that same process for endpoints marked "Standard" in the table above:

  • Data that has never been submitted is never returned - unless you're an admin (see below).
  • Submitted-but-not-yet-reviewed data is returned only if you pass includeUnreviewed=1, or if you're an admin.
  • Reviewed data is always returned (subject to your other filters).

A row belongs to exactly one field test, which belongs to exactly one site (or, for field-observation-based data, one field observation, which likewise belongs to one site) - so this check follows that chain up to the site automatically. You don't need to (and can't) filter on review status directly.

Files is the one exception: a single file can be attached to a site, a field test, a field observation, and/or a sample all at once. A file is returned only if every one of the things it's attached to passes the check above - it's hidden as soon as any single attachment isn't submitted/reviewed, even if others are.

Endpoints marked "None" in the tables above (Events, Stations, Segments, Intensity measures, ground motion types/components, Citations, Labs, Lab programs, and the Lab programs ↔ Citations junction) are shared reference data with no single owning site, so every authenticated request sees every row - includeUnreviewed has no effect on them.

The other junction endpoints follow whichever side of the join actually has a review workflow - e.g. Lab programs ↔ Samples is gated by the Sample's field test/site even though Lab programs themselves have no STAT/REVW of their own; a row is hidden as soon as that side fails the check, same as any other "Standard" endpoint.

Admins: if your account has admin permissions, pass role=admin to bypass review-status filtering entirely - every row is returned regardless of STAT or REVW, including data that's never been submitted at all. This has no effect on endpoints marked "None" above, since they're already unfiltered. If you pass role=admin without actually having admin permissions, the request fails with an error rather than silently falling back to normal filtering - and admin status only ever applies when you explicitly ask for it with role=admin; it's never assumed just because you happen to be an admin.

Example Requests

Example 1 - the 20 southernmost reviewed sites:

https://nextgenerationliquefaction.org/sites/api-index?limit=20&sort=SITE_LAT&direction=asc

Example 2 - field tests of a couple of specific types, including unreviewed ones:

https://nextgenerationliquefaction.org/field-tests/api-index?where=TEST_TYPE IN (BORH,SCPG)&includeUnreviewed=1

Example 3 - only a couple of fields, filtered by a range:

https://nextgenerationliquefaction.org/sites/api-index?select=SITE_ID,SITE_NAME,SITE_LAT,SITE_LON&where=SITE_LAT>30 AND SITE_LAT<40

Full Python example, using the token from Authentication above:

import pandas as pd
import json

headers = {}
headers["Accept"] = "application/json"
headers["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) 134.0.6998.118 Safari/537.36"
headers["Authorization"] = "Bearer {}".format(token)
url = 'https://nextgenerationliquefaction.org/field-observations/api-index?limit=5000&includeUnreviewed=0'
r = requests.get(url, headers=headers)
df = pd.DataFrame.from_dict(json.loads(r.text))

Possible Problems

  • Empty [] when you expected data. Most often this means the data hasn't been reviewed yet - try adding includeUnreviewed=1 (this has no effect on the reference-data endpoints, which always return everything).
  • 400 response with an errorMessage. The message explains what's wrong (bad/expired token, unknown field name, invalid operator for a field's type, etc.) - note that it may contain literal <br> tags between multiple errors rather than newlines.
  • A request that returns fewer rows than limit. That's the last page - see the pagination note under Query Parameters.
  • Large tables. Some endpoints (e.g. cone penetration data) have hundreds of thousands of rows - filter with where= and/or select= rather than pulling an entire table at once.