API Documentation

The Insights API provides access to Property data for US properties. The Insights API follows OData protocol with the addition of custom querying options.

Authentication

Access to the API utilizes OAuth2.

Client Credentials (ID and secret) are provided for each user of the API, allowing for retrieval of bearer access token with preset expiry (default 3600 seconds).

We must encode the combination of Client ID and Client Secret Key separated by ":" in Base 64 format like this:

Base64Encode(5pq9999coididi613e222o1nnpp: fhkoq33a69d8191j1tercv35037clb9e5a7d215e64e4e) = 85e1b0e7539a4a96b00ee02b335aa6418c580917a6b4667f6f7a6fe2149536041569924dfbe4a7df33f==

Can use online service Base64Encode for the encoding.

The end point for the token must include two parameters:

1. "grant_type" = "client_credentials"
2. "client_id" = "5pq9999coididi613e222o1nnpp"

Example request to retrieve access token:

curl -X POST -H "Content-Type: application/x-www-form-urlencoded" \
-H "Authorization: Basic 85e1b0e7539a4a96b00ee02b335aa6418c580917a6b4667f6f7a6fe2149536041569924dfbe4a7df33f==" \
-H "Host: authenticate.constellation1apis.com" \
"https://authenticate.constellation1apis.com/oauth2/token?grant_type=client_credentials&client_id=5pq9999coididi613e222o1nnpp"

Authentication response:

{
"access_token": "**********************************",
"expires_in": 3600,
"token_type": "Bearer"
}

Usage

Once the access token is retrieved, it is needed in each call to the CDatalabs API under the Authorization HTTP header:

curl -X GET -i -H "Host: insights.cdatalabs.com" \
-H "Authorization: Bearer **********************************" \
"https://insights.cdatalabs.com/odata/Assessor?$top=5"

Recommendation: It is recommended to include 'Accept-Encoding' header with value 'gzip, deflate, br'. This will compress the response into a zip/brotli format. This should improve the latency in transit by compressing the size. Client will have to decompress the response before processing it:

Metadata

The Insights API has an endpoint to retrieve metadata:

https://insights.cdatalabs.com/odata/$metadata

This endpoint allows caller to know available fields and data types for each entity.

<?xml version="1.0" encoding="utf-8" ?>
<edmx:Edmx Version="4.0" xmlns:edmx="http://docs.oasis-open.org/odata/ns/edmx">
    <edmx:DataServices>
        <Schema Namespace="Property.Api.Snowflake.Model" xmlns="http://docs.oasis-open.org/odata/ns/edm">
            <EntityType Name="Assessor" />    
            <EntityType Name="Assignment" />
            <EntityType Name="Avm" />
            <EntityType Name="Deed" />
            <EntityType Name="Mortgage" />
            <EntityType Name="PreForeclosure" />
            <EntityType Name="Release" />
            <EntityType Name="School" />
            <EntityType Name="SchoolReviews" />
            <EntityType Name="SchoolRanksAndDemographics" />
            <EntityType Name="SchoolTestScores" />
            <EntityType Name="LoanOriginatorAnalytics" />
	    <EntityType Name="PointofInterest" />
	    <EntityType Name="Neighborhood Community" />
	    <EntityType Name="Household Demographic" />
	    <EntityType Name="Individual Demographic" />
	    <EntityType Name="Transit Data" />
        </Schema>
    </edmx:DataServices>
</edmx:Edmx>

Entities

The Insights API supports several entity types, primarily Assessor which represents a Property tax assessment. Other entity types such as Deed represents property ownership records.

Basic Requests

Following OData protocol, the following examples show how combining options and filters can retrieve a desired subset of data results.

GET /odata/Assessor?$select=...&$filter=...&$top=...&$orderby=...&$skip=...

API response structure

    {
    "@odata.context": "...",
    "@odata.total_count": 100,
    "@odata.count": 0,
    "@odata.nextLink": "...",
    "value": []
    }

Select

By default, the entire set of fields are returned, however, we can whitelist specific fields in the response using $select. Whitelist fields are specified in a comma separated list.

Example request:

GET /odata/assessor?$select=CountyFipsCode,TotalAssessedValue

Example response:

    {
    "@odata.context": "https://insights.cdatalabs.com/$metadata#Assessor",
    "@odata.total_count": 100,
    "@odata.count": 10,
    "@odata.nextlink": "https://insights.cdatalabs.com/odata/assessor/12wqewqeqqxcz4ewogICJGcm9tIjogMTAwLAogICJTaXplIjogMTAwLAogICJUaW1lU3RhbXAiOiA2Mzg1MzA5MTkzMDYzNzA3NjIKfQ==/?$select=CountyFipsCode,TotalAssessedValue"
    "value": [
    {
    "CountyFipsCode": "35043",
    "TotalAssessedValue": 86516
    },
    {
    "CountyFipsCode": "35001",
    "TotalAssessedValue": 103504
    }
    ...
    ]
    }

Top

By default, the first 100 results are returned in any API request, however we can alter the number of results by using the $top option followed by a non-negative integer value.

Example request:

GET /odata/Assessor?$top=1

Recommendation: It is recommended to not exceed $top=100. Exceeding this limit will cause slowness of response payloads

Skip

Often in combination with $top, the $skip option allows for client side paging, also using a non-negative integer we can cycling through the entire result set.

Example request:

GET /odata/Assessor?$top=5&skip=2

Recommendation: It is recommended to use this approach for shallow (small dataset) paging. Using this approach for more than 1 million records will cause slowness in response.

SkipToken

When working with large datasets, our OData API supports pagination to efficiently manage the data retrieval process. This is achieved using the @odata.nextLink property, which provides a URL to the next page of results.

Initial request:

Begin by making a GET request to the desired endpoint. The response will include the first set of results along with '@odata.nextLink' property if more data is available.

Example initial request:

GET /odata/Assessor?$expand=Deeds&$filter=StateOrProvince eq 'CA'

Recommendation: It is recommended to use this approach for paging large datasets.

Checking for '@odata.nextLink'

After receiving the initial response, inspect the JSON payload for the @odata.nextLink property. This property contains the URL for the next page of results. If this property is present, it indicates that there are additional pages of data to retrieve.

Sample JSON response:

    {
    "@odata.context": "https://insights.cdatalabs.com/$metadata#Assessor",
    "@odata.total_count": 100,
    "@odata.count": 100,
    "@odata.nextlink": "https://insights.cdatalabs.com/odata/Assessor/skiptoken/ewogICJGcm9tIjogMTAwLAogICJTaXplIjogMTAwLAogICJUaW1lU3RhbXAiOiA2Mzg1MzE4ODQ2NDM4NzQ4NTYKfQ==/?$expand=Deeds&$filter=StateOrProvince+eq+'CA'",
    "value": [...]
    }

Fetching Subsequent Pages

To retrieve the next set of results, make another HTTP GET request to the URL specified in the '@odata.nextLink' property. Repeat this process, checking each response for a new @odata.nextLink, until this property is no longer present.

Example of fetching next page:

GET https://insights.cdatalabs.com/odata/Assessor/skiptoken/ewogICJGcm9tIjogMTAwLAogICJTaXplIjogMTAwLAogICJUaW1lU3RhbXAiOiA2Mzg1MzE4ODQ2NDM4NzQ4NTYKfQ==/?$expand=Deeds&$filter=StateOrProvince+eq+'CA'

Looping Until All Data is Retrieved

Continue the pagination process in a loop until no @odata.nextLink is found. This ensures that all pages of data are retrieved sequentially.

Best Practices

1. Error Handling: Implement robust error handling to manage network issues, API rate limits, and other potential errors.
2. Data Processing: Consider processing each page of data as it is fetched to avoid excessive memory usage.
3. Respect Rate Limits: Be mindful of the API's rate limits to prevent throttling or blocking.

Filter

Most common option is $filter, typically at least one filter is used when retrieving data, often used are comparison operators to filter results by an MLS (Data Source) via the OriginatingSystemName field.

The following is a list of supported operators and expressions. In future additional OData expressions could be added to the API.

Comparison Operators

Note: single quotation is required for string values

Boolean Operators

Note: boolean operator is expressed in lower cased

Often used to chain multiple comparison expressions, the boolean operators and and or can precise subset of data

Expand

In OData, the $expand allows for retrieval of supplementary entity data where there is an association to the primary entity. Typically $expand would be used where Property is the primary EntityType. Expand can retrieve multiple sub-entities using a comma separated list.

Example requests:

GET /odata/Assessor?$expand=Deeds
GET /odata/Assessor?$expand=Deeds,Mortgage

Recommendation: It is recommended to not exceed $top=100 with $expand as $expand increases the payload size and query time, potentially resulting in a degradation of service.

Ordering

To sort results, the $orderby option can be used, providing the desired sort field and direction, if direction is not provided by default it is assumed the sort order is ascending. Multiple order by expressions can be provided by a comma separated list.

Example requests:

GET /odata/Assessor?$orderby=CID
GET /odata/Assessor?$orderby=CID desc
GET /odata/Assessor?$orderby=CID desc, AssessedLandValue asc

Schools

These set of tables contain detailed data about various schools, including their rankings, addresses, total student populations, and the origins of these students. The data also encompasses academic performance metrics such as grades and test scores. This information provides a holistic view of each school's demographic and academic standing, enabling a thorough analysis of their performance and the diversity of their student bodies.

School

This dataset consists of current information for a current school and school district includes private and public school details, such as demographics and test scores. This data covers overview of various aspects like school demographics, performance metrics, and the academic ranking of schools.

School Demographics and Ranks

This dataset includes historical information spanning the years 2012 to previous year about schools, covering both private and public institutions. This data encompasses school rankings, graduation years, and spending per pupil.

School Reviews

School and school district reviews include both historical and current data. These reviews feature comments from parents, teachers, and students, along with rating stars, the date of the review, and the specific School and School district information it pertains to.

School Test Scores

Schools and School Districts test scores, both historical and current year, are included.

Sample Queries

School is the basic dataset which can be queried using the /odata/School endpoint:

GET /odata/School?$filter=AddressState eq 'TX'

State is the basic unit of geography to lookup schools

We can further filter it down to Local Education Authority ('FIPSLeaId')

GET /odata/School?$filter=AddressState eq 'TX' and FIPSLeaId eq '23640'

A US State can comprise of more than one Local Education Authority

If we want to find one school in a particular Local Education Authority, we can filter on the FIPSSchoolId field

GET /odata/School?$filter=AddressState eq 'TX' and FIPSLeaId eq '23640' and FIPSSchoolId eq '02530'

Expand

Like other datasets, we can expand SchoolReviews, SchoolTestScores, and SchoolRandAndDemographics from School

GET /odata/School?$filter=AddressState eq 'TX' and FIPSLeaId eq '23640' and FIPSSchoolId eq '02530'&$expand=SchoolReviews,SchoolTestScores,SchoolRanksAndDemographics

Private Schools

Some shools are not affiliated to any Local Education Authority. These schools have FIPSLeaId value: '99999'

GET odata/School?$filter=FIPSLeaId eq '99999' and AddressState eq 'TX'

These Schools also have their 'SchoolIsPrivate' field value true

Valuation

This dataset contains comprehensive property information, including valuation details, ownership records, and key property attributes. It provides insights into residential real estate, covering property location, lot size, square footage, number of bedrooms and bathrooms, and year built. Additionally, the dataset includes sales history, estimated market value, confidence scores, and price range estimates, offering valuable insights for real estate analysis and investment decisions.

Sample Queries

Valuation is the basic dataset which can be queried using the /odata/Valuation endpoint:

GET /odata/Valuation?Address=1316 CHELSEA WAY&City=Stockton&State=CA&Zip=95209

GET /odata/Valuation?Zip=95209

GET /odata/Valuation?PropertyId=157610188

A result code of 0 means a single match with valuation details included.

A result code of 11 means multiple matches were found, but valuation details are not included.

Either a ZIP code or a Property ID is required for accurate property valuation lookup. Including an address, city, and state helps refine the search, ensuring more precise results.

API Quotas

APIs can be set on quota usage subscriptions. These are the types of quotas:

1. API requests on monthly volume
2. API response record counts (number of records per month)
3. API response cumulative data size (cumulative data size fetched per month)
4. API response size limit (10MB response payload limit)

Response Descriptions

* -> the numbers in the table above are examples.