Release Notes
Find out what the Engineering team has been up to across our entire platform, including general announcements, updates, and bugfixes.
⚠️ Final Reminder:
Token-based pagination has been operational for the past two months. Since page-based pagination will be discontinued at the start of August, this is your final month to update your integrations.
✨ New Permit Records:
-
Total permit records: 171,752,486
That’s up from 170,626,720, a jump of over 1.1 million!
-
June additions: Over 25,000 new records from June alone.
-
New construction permits:
- 150,000+ new construction records
- 50,000 of which are solar projects
🚀 Expanded Jurisdiction Coverage:
We’ve added dozens of new jurisdictions, some of which are:
- Livingston Parish, LA – 6,951 new permits
- Rapides Parish, LA – 4,454 new permits
- Dallas County, TX – 706 new permits
- Tequesta, FL – 320 new permits
🧑🏼🔧 Contractor Data Enhancements:
- 14,000 new contractor records added
- Nearly 20,000 new primary phone numbers
- Over 20,000 new primary email addresses
🚀 Upgrades:
- Faster pagination, especially for deeper pages.
- More jurisdiction coverage.
🚀 Upgrades:
- Faster pagination, especially for deeper pages.
- More jurisdiction coverage.
✨ Reminder:
- Added cursor-based pagination for all endpoints, which substantially improves deep search pagination.
This replaces the former page
parameter for pagination. This old method will be supported for an additional three months, at which point it will be deprecated.
✨ New:
- Added 1.2 million permits, including 248,000 from May alone, significantly boosted by our new infrastructure.
- 300 new jurisdictions are now online, increasing our nationwide coverage.
- 70,000 new contractors added, bringing the total to 2.96 million.
- Enhanced deduplication logic eliminated duplicate contractor records, improving data quality.
- 40,000 additional contact details (phone numbers, emails, addresses) were linked, thanks to infrastructure enhancements.
⚠️ Breaking Changes & Compatibility Notes
‼️ Permit ID Updates:
- Approximately 2 million duplicate permits from legacy system migrations were removed. The most affected states are Florida and Texas.
- ~6% of permits now have new IDs. Attributes such as
permit_number
and address remain unchanged.
If you query by cached id
and encounter missing results (404), use permit_number
, address, or other metadata to re-fetch the permit. If you still encounter issues, reach out—we’ll help map old IDs to new ones.
‼️ Contractor ID Updates
- ~170,000 contractors have new IDs that have been regenerated based on improved data quality.
- All contractors remain searchable by:
- Areas of work
- Permit projects
- Business names
If you cache contractor IDs on your end and you encounter not-found responses, use other record attributes to update IDs on your end. If ID changes affect your workflow in a more substantial way, contact us directly and we’ll assist.
🚀 Upgrades:
- Improved the Permit Search logic to always order resulting permits by newest first.
🚀 Upgrades:
- Improved the Permit Search logic to always order resulting permits by newest first.
✨ New:
- Added cursor-based pagination for all endpoints, which substantially improves deep search pagination.
This replaces the former page
parameter for pagination. This old method will be supported for an additional three months, at which point it will be deprecated.
🚀 Upgrades:
- Improved the response ordering logic to always order results in chronological order, descending.
✨ New:
- Added a new
apn
column to theaddresses
table - Added two new columns
status
andstatus_detailed
to thecsl
andcontractors
tables- Status codes for all states are mapped against the following schema:
active
: License is active and in good standing with no restrictions.active_conditional
: License is active but subject to specific conditions, limitations, or temporary provisions.expired
: License has lapsed due to non-renewal; typically eligible for renewal.retired
: License has been voluntarily placed in an inactive state, often due to the holder retiring or the business closing.suspended
: License is temporarily invalid, usually due to disciplinary actions or unresolved administrative issues.revoked
: License has been permanently terminated, typically due to serious violations or disciplinary reasons.pending
: License application, renewal, or reinstatement is currently under review or being processed.inactive
: License is currently not valid, often due to administrative issues or other undefined reasons not captured by the categories above.
- Status codes for all states are mapped against the following schema:
🚀 Upgrades:
- Expanded
csl
table coverage by adding 11 new states:CT
,GA
,IL
,MD
,MO
,OK
,NY
,OR
,SC
,TN
,UT
.
🐞 Bugfix:
- Fixed an address parsing issue that was affecting
csl
table contractors inNV
.
✨ New:
- General speed improvements
- New and improved onboarding flow
🪲 Bug Fixes:
- Fixed bug that didn’t change the geo on the download list button
- Fixed bug preventing some filters from being “sticky” in between searches
- Fixed bug preventing geography profile filters from updating charts
✨ New:
- General speed improvements
- New and improved onboarding flow
🪲 Bug Fixes:
- Fixed bug that didn’t change the geo on the download list button
- Fixed bug preventing some filters from being “sticky” in between searches
- Fixed bug preventing geography profile filters from updating charts
No API changes this month!
🏗️ Permits:
- Total permits: The dataset now contains 171M permits — an increase of 2M compared to the previous batch. With 8 new jurisdictions.
first_seen_date
coverage: ~150M permits have a first_seen_date
🧑🏼🔧 Contractors:
- We now have over 3 million contractors in the dataset, with ~50K new records added this month
- License linkage: An additional 80K contractors have been linked to state license files.
📇 Classification coverage:
- All contractors with a license now have the
classification_derived
field as well. This field maps state-specific classifications into standardized categories. - If a contractor has multiple license classifications, they are concatenated using the pipe (”|”) character.
🗂️ License Files:
- A new table, CSL, is added to our production database. It contains data from 26 states files, and has a total of 1.8 M contractors.
- Around 200K contractors are linked to our Contractors table while the others are new. As part of this new dataset, we have around 700K new contractors phone numbers and 200K new emails.
There were in total almost 3000 different classifications categories across different state files. We standardized this to the following categories:
concrete_and_paving
, demolition_and_excavation
, electrical
, fencing_and_glazing
, framing_and_carpentry
, general_building_contractor
, general_engineering_contractor
, hvac
, landscaping_and_outdoor_work
, plumbing
, roofing
, specialty_trades
✨ New:
- Added Website Search filter options to Contractor Filters
- Added tooltips to the UI to better explain functionality
🚀 Upgrades:
- Added Search filter auto-caching to improve UX between page reloads and sessions
- Added new New User onboarding experience
- Adjusted datepicker widget for date range selection to monthly, which will improve usability
✨ New:
- Added Website Search filter options to Contractor Filters
- Added tooltips to the UI to better explain functionality
🚀 Upgrades:
- Added Search filter auto-caching to improve UX between page reloads and sessions
- Added new New User onboarding experience
- Adjusted datepicker widget for date range selection to monthly, which will improve usability
✨ New:
-
Added new fields to the
/contractors/
endpoint group:New `/contractors/` fields
first_seen_date
: Date when the contractor was first recorded in the Shovels platform and given an id.license_act_date
: Date when the contractor’s license became active.license_inact_date
: Date when the contractor’s license became inactive.review_count
: Number of reviews the contractor has received.rating
: Rating of the contractor based on reviews.dba
: “Doing Business As” name for the contractor.sic
: Standard Industrial Classification (SIC) code of the contractor.naics
: North American Industry Classification System (NAICS) code of the contractor.linkedin_url
: LinkedIn URL of the contractor.revenue
: Annual revenue of the contractor’s business.employee_count
: Number of employees working for the contractor.primary_industry
: Primary industry in which the contractor operates.
-
Implemented new API rate limits: 1M requests/month, or ~33k requests/day.
✨ New:
- Added
first_seen_date
to thepermits
table- this represents the date when the the permit itself was first retrieved for the Shovels platform
This is different than the first_seen_date
column in the contractors
table, which by contrast represents the date the contractor was first seen in our platform and assigned an id
.
FINAL WARNING: As previously announced, the V1 of the Shovels API is now deprecated.
✨ New:
- Added Address Profiles
- Added Free Forever plan in place of a Free Trial.
🐞 Bugfix:
- Fixed issue where Geography Profiles (
City
,Jurisdiction
,County
, etc) filters for Property Type and Permit Category weren’t updating counts correctly.
✨ New:
- Added Address Profiles
- Added Free Forever plan in place of a Free Trial.
🐞 Bugfix:
- Fixed issue where Geography Profiles (
City
,Jurisdiction
,County
, etc) filters for Property Type and Permit Category weren’t updating counts correctly.
✨ New:
- Added new
/v2/{geography}/{geo_id}/metrics/monthly
endpoint. - Added new
/v2/{geography}/{geo_id}/metrics/current
endpoint.
💥 Breaking:
-
Removed
/v2/{geography}/{geo_id}/metrics
endpoint. -
Changed monetary value data types from
decimal
integer
(eg,dollars
cents
). This will affect the following parameters:/permits
property_assess_market_value
job_value
fees
/contractors
avg_job_value
total_job_value
/contractors/{id}/metrics
avg_job_value
total_job_value
/{geography}/{geo_id}/metrics/{interval}
total_job_value
-
Changed percentage data types from
decimal
integer
(eg,0.75
75
). This will affect the following parameters:/permits
inspection_pass_rate
/contractors
avg_inspection_pass_rate
/contractors/{id}/metrics
avg_inspection_pass_rate
/{geography}/{geo_id}/metrics/{interval}
avg_inspection_pass_rate
🐞 Bugfix:
- Fixed issue where
address
object’s latitude and longitude coordinates were occasionally reversed, and built in anomaly detection and correction for future processing.- This fix continues downstream to API and Online records.
Breaking Changes Policies
Breaking Changes Policies
In order to ensure smooth transitions between updates, we’ve settled on a few short term policies.
- We will do our best to give as much advance warning on breaking changes as possible.
- Some recent breaking changes are delayed updates to the schema that should have been part of the V2 launch. We do not plan on making breaking changes this often.
- Where sensible and possible, we will provide backward compatibility for new endpoints, parameters, and schemas.
Address Coordinate Errors
Address Coordinate Errors
As described under EDL changes above, we found that there were rare cases where the latitude and longitude coordinates were reversed. Affected records have been corrected.
We’ve also beefed up our address-geospatial processing to find and fix these errors going forward.
Effective Dates
Effective Dates
- Online: Monday, February 03, 2025.
- API: Monday, February 10, 2025.
- EDL: Wednesday, February 6, 2025.
✨ New:
- Added
employees
to Contractor Profiles. - Added links to Jurisdiction Profiles for all permits.
- Added links to City Profiles and County Profiles for all geo-related profiles.
🐞 Bugfixes:
- Fixed password reset bug.
✨ New:
- Added
employees
to Contractor Profiles. - Added links to Jurisdiction Profiles for all permits.
- Added links to City Profiles and County Profiles for all geo-related profiles.
🐞 Bugfixes:
- Fixed password reset bug.
✨ New:
- Added new
/v2/contractors/{id}/employees
endpoint. - Added new
/v2/addresses/{id}/metrics
endpoint.
⚠️ End of Life:
- We will DEPRECATE the API V1 by the end of January 2025.
Please update all endpoints using the old schema before January 31, 2025 to ensure continued usage.
🐞 Bugfixes:
- Fixed metrics calculation algorithm on all endpoints.
💥 Breaking:
contractors
table:first_seen_at
renamedfirst_seen_date
.- Data type changed
DATE
.
- Data type changed
reviews
renamedreview_count
.
residents
table:- Foreign key changed from
permit_id
address_id
.- Please note: The
address_id
field is also the foreign key for thepermits
table, allowing easier joins betweenresidents
andpermits
tables.
- Please note: The
- Foreign key changed from
Please ensure that any existing queries or automation using first_seen_at
, reviews
, or permit_id
are updated accordingly.
Coverage Expansion
Coverage Expansion
These new permits will be available across the entire platform.
✨ New:
- +6M permits nationwide.
- +9 permit jurisdictions.
✨ New:
- Added +2M permits in
Texas
. - Added metric visualizations to Contractor Profiles.
🐞 Bugfixes:
- Corrected issue where
Contractor Profiles
wouldn’t load properly.
✨ New:
- Added +2M permits in
Texas
. - Added metric visualizations to Contractor Profiles.
🐞 Bugfixes:
- Corrected issue where
Contractor Profiles
wouldn’t load properly.
✨ New:
residents
endpoint.
🚀 Upgrades:
- Improved address validation for all geography-related fields, in line with US Census data.
💥 Breaking:
- Casing standardizations (please update accordingly if your logic is case-sensitive):
property_type
values are now lower case, egresidential
.
✨ New:
first_seen_at
column in thecontractor
table.employees
table, linked tocontractors
table:- Includes a wide range of firmographic data for individual employees.
- See “Schema Updates” below for the full list of columns.
🚀 Upgrades:
- additional columns in the
residents
table. See “Schema Updates” below for the full list.
💥 Breaking:
- Casing standardizations (in addition to the changes outlined in the API Section previously).
property_owner_type
values are now snake_case, egindividual
orcompany_owned
.owner_name
,owner_street
,owner_city
values are now upper case, egJANE DOE
,MAIN ST
, orAGAWAM
.applicant_name
,applicant_street
,applicant_city
values are now upper case, just likeowner_$
above.
- (PLANNED) Replacement of
permits_ids
withaddress_id
in theresidents
table (please update your pipelines that use this foreign key value.)- This change will go into effect January 2025.
Schema Updates
Schema Updates
We made a number of additions to our datasets, including new columns. For breaking changes to casing, see “API” above.
New columns in the 'residents' table:
New columns in the 'residents' table:
personal_emails_validation_status
(string): The validation status of the associated personal email. ‘Valid’ indicates a validated email; null means unknown.personal_emails_last_seen
(date): The date of the last validation or verification attempt for the personal email. Null if unknown.business_email
(string): The primary business email observed for this person.business_email_validation_status
(string): Validation status of the business email. ‘Valid’ means the email was validated; null means unknown. May contain values like ‘Valid-ESP’ (validated by email service provider) or ‘Valid-Digital’ (validated by cookie/digital tag).business_email_last_seen
(date): The date of the last known validation or verification attempt of the business email. Null if unknown.linkedin_url
(string): URL of the person’s LinkedIn profile.homeowner
(string): Reports if the person in this record is a homeowner. Y and N are observed values, P represents that they are likely a homeowner, based on probabilistic modeling, and null values represent ‘unknown’.gender
(string): The person’s gender.age_range
(string): The person’s age mapped to standard demographic ranges.is_married
(boolean): Indicates if the person is married.has_children
(boolean): Indicates if the person has children.income_range
(string): The person’s income range.net_worth
(string): The person’s net worth, mapped to standard demographic ranges. Null if unknown.job_title
(string): The person’s job title.seniority_level
(string): The seniority level of the person’s role.department
(string): The department in which the person works.job_title_last_updated
(date): The last date the person’s job title was updated.last_updated
(date): The last date any value in this record was updated.work_history
(string, JSON): A JSON-formatted history of the person’s work experiences (e.g., company name, * position, duration, start/end time, job description, location, social_url).education_history
(string): The person’s education background.social_connections
(string): The number of social media connections the person has, provided in ranges (e.g., 1-9, 10-49, etc.).company_name
(string): The name of the company where the person works.company_domain
(string): The company’s domain.company_phone
(string): The company’s contact phone number.company_sic
(string): The Standard Industrial Classification (SIC) code(s) of the company, separated by semicolons if multiple.company_street_no
(string): The street number of the company’s address.company_street
(string): The street name of the company’s address.company_city
(string): The city in which the company is located.company_zip
(string): The company’s 5-digit ZIP code.company_state
(string): The uppercase state abbreviation for the company’s address.company_linkedin_url
(string): The URL of the company’s LinkedIn profile.company_revenue
(string): The company’s revenue, expressed in standard ranges.company_employee_count
(string): The number of observed US enterprise employees at the company, provided in * standard firmographic ranges.company_primary_industry
(string): The primary industry in which the company operates.company_description
(string): A description of the company’s business activities.company_naics
(string): The company’s North American Industry Classification System (NAICS) code(s).
New 'employees' table (connected to 'contractors'):
New 'employees' table (connected to 'contractors'):
contractor_id
(string)person_id
(string)name
(string)street_no
(string)street
(string)city
(string)zipcode
(string)zipcode_ext
(string)state
(string)phone
(string)email
(string)email_validation_status
(string)email_last_seen
(date)business_email
(string)business_email_validation_status
(string)business_email_last_seen
(date)linkedin_url
(string)homeowner
(string)gender
(string)age_range
(string)is_married
(boolean)has_children
(boolean)income_range
(string)net_worth
(string)job_title
(string)seniority_level
(string)department
(string)job_title_last_updated
(date)work_history
(string, JSON formatted)education_history
(string)