Release Notes

​

🚀 Upgrades:

• Minor improvements to the UX and ongoing speed enhancements.

​

🐞 Bug Fixes:

• Bug fixes

​

⚠️ Final Reminder:

Token-based pagination has been operational for the past two months. Since page-based pagination will be discontinued at the start of August, this is your final month to update your integrations.

​

✨ New Permit Records:

• Total permit records: 171,752,486

  That’s up from 170,626,720, a jump of over 1.1 million!

• June additions: Over 25,000 new records from June alone.

• New construction permits:

  • 150,000+ new construction records
  • 50,000 of which are solar projects

​

🚀 Expanded Jurisdiction Coverage:

We’ve added dozens of new jurisdictions, some of which are:

• Livingston Parish, LA – 6,951 new permits
• Rapides Parish, LA – 4,454 new permits
• Dallas County, TX – 706 new permits
• Tequesta, FL – 320 new permits

​

🧑🏼‍🔧 Contractor Data Enhancements:

• 14,000 new contractor records added
• Nearly 20,000 new primary phone numbers
• Over 20,000 new primary email addresses

​

🚀 Upgrades:

• Faster pagination, especially for deeper pages.
• More jurisdiction coverage.

​

✨ Reminder:

• Added cursor-based pagination for all endpoints, which substantially improves deep search pagination.

​

✨ New:

• Added 1.2 million permits, including 248,000 from May alone, significantly boosted by our new infrastructure.
• 300 new jurisdictions are now online, increasing our nationwide coverage.
• 70,000 new contractors added, bringing the total to 2.96 million.
• Enhanced deduplication logic eliminated duplicate contractor records, improving data quality.
• 40,000 additional contact details (phone numbers, emails, addresses) were linked, thanks to infrastructure enhancements.

⚠️ Breaking Changes & Compatibility Notes

​

‼️ Permit ID Updates:

• Approximately 2 million duplicate permits from legacy system migrations were removed. The most affected states are Florida and Texas.
• ~6% of permits now have new IDs. Attributes such as permit_number and address remain unchanged.

​

‼️ Contractor ID Updates

• ~170,000 contractors have new IDs that have been regenerated based on improved data quality.
• All contractors remain searchable by:
  • Areas of work
  • Permit projects
  • Business names

​

🚀 Upgrades:

• Improved the Permit Search logic to always order resulting permits by newest first.

​

✨ New:

• Added cursor-based pagination for all endpoints, which substantially improves deep search pagination.

​

🚀 Upgrades:

• Improved the response ordering logic to always order results in chronological order, descending.

​

✨ New:

• Added a new apn column to the addresses table
• Added two new columns status and status_detailed to the csl and contractors tables
  • Status codes for all states are mapped against the following schema:
    • active: License is active and in good standing with no restrictions.
    • active_conditional: License is active but subject to specific conditions, limitations, or temporary provisions.
    • expired: License has lapsed due to non-renewal; typically eligible for renewal.
    • retired: License has been voluntarily placed in an inactive state, often due to the holder retiring or the business closing.
    • suspended: License is temporarily invalid, usually due to disciplinary actions or unresolved administrative issues.
    • revoked: License has been permanently terminated, typically due to serious violations or disciplinary reasons.
    • pending: License application, renewal, or reinstatement is currently under review or being processed.
    • inactive: License is currently not valid, often due to administrative issues or other undefined reasons not captured by the categories above.

​

🚀 Upgrades:

• Expanded csl table coverage by adding 11 new states: CT, GA, IL, MD, MO, OK, NY, OR, SC, TN, UT.

​

🐞 Bugfix:

• Fixed an address parsing issue that was affecting csl table contractors in NV.

​

✨ New:

• General speed improvements
• New and improved onboarding flow

​

🪲 Bug Fixes:

• Fixed bug that didn’t change the geo on the download list button
• Fixed bug preventing some filters from being “sticky” in between searches
• Fixed bug preventing geography profile filters from updating charts

No API changes this month!

​

🏗️ Permits:

• Total permits: The dataset now contains 171M permits — an increase of 2M compared to the previous batch. With 8 new jurisdictions.
• first_seen_date coverage: ~150M permits have a first_seen_date

​

🧑🏼‍🔧 Contractors:

• We now have over 3 million contractors in the dataset, with ~50K new records added this month
• License linkage: An additional 80K contractors have been linked to state license files.

​

📇 Classification coverage:

• All contractors with a license now have the classification_derived field as well. This field maps state-specific classifications into standardized categories.
• If a contractor has multiple license classifications, they are concatenated using the pipe (”|”) character.

​

🗂️ License Files:

• A new table, CSL, is added to our production database. It contains data from 26 states files, and has a total of 1.8 M contractors.
• Around 200K contractors are linked to our Contractors table while the others are new. As part of this new dataset, we have around 700K new contractors phone numbers and 200K new emails.

​

✨ New:

• Added Website Search filter options to Contractor Filters
• Added tooltips to the UI to better explain functionality

​

🚀 Upgrades:

• Added Search filter auto-caching to improve UX between page reloads and sessions
• Added new New User onboarding experience
• Adjusted datepicker widget for date range selection to monthly, which will improve usability

​

✨ New:

• Added new fields to the /contractors/ endpoint group:

  New `/contractors/` fields

• Implemented new API rate limits: 1M requests/month, or ~33k requests/day.

​

✨ New:

• Added first_seen_date to the permits table
  • this represents the date when the the permit itself was first retrieved for the Shovels platform

​

✨ New:

• Added Address Profiles
• Added Free Forever plan in place of a Free Trial.

​

🐞 Bugfix:

• Fixed issue where Geography Profiles (City, Jurisdiction, County, etc) filters for Property Type and Permit Category weren’t updating counts correctly.

​

✨ New:

• Added new /v2/{geography}/{geo_id}/metrics/monthly endpoint.
• Added new /v2/{geography}/{geo_id}/metrics/current endpoint.

​

💥 Breaking:

• Removed /v2/{geography}/{geo_id}/metrics endpoint.

• Changed monetary value data types from decimal integer (eg, dollars cents). This will affect the following parameters:

  /permits

  /contractors

  /contractors/{id}/metrics

  /{geography}/{geo_id}/metrics/{interval}

• Changed percentage data types from decimal integer (eg, 0.75 75). This will affect the following parameters:

  /permits

  /contractors

  /contractors/{id}/metrics

  /{geography}/{geo_id}/metrics/{interval}

​

🐞 Bugfix:

• Fixed issue where address object’s latitude and longitude coordinates were occasionally reversed, and built in anomaly detection and correction for future processing.
  • This fix continues downstream to API and Online records.

​

✨ New:

• Added employees to Contractor Profiles.
• Added links to Jurisdiction Profiles for all permits.
• Added links to City Profiles and County Profiles for all geo-related profiles.

​

🐞 Bugfixes:

• Fixed password reset bug.

​

✨ New:

• Added new /v2/contractors/{id}/employees endpoint.
• Added new /v2/addresses/{id}/metrics endpoint.

​

⚠️ End of Life:

• We will DEPRECATE the API V1 by the end of January 2025.

​

🐞 Bugfixes:

• Fixed metrics calculation algorithm on all endpoints.

​

💥 Breaking:

• contractors table:
  • first_seen_at renamed first_seen_date.
    • Data type changed DATE.
  • reviews renamed review_count.
• residents table:
  • Foreign key changed from permit_id address_id.
    • Please note: The address_id field is also the foreign key for the permits table, allowing easier joins between residents and permits tables.

​

✨ New:

• Added +2M permits in Texas.
• Added metric visualizations to Contractor Profiles.

​

🐞 Bugfixes:

• Corrected issue where Contractor Profiles wouldn’t load properly.

✨ New:

• residents endpoint.

​

🚀 Upgrades:

• Improved address validation for all geography-related fields, in line with US Census data.

​

💥 Breaking:

• Casing standardizations (please update accordingly if your logic is case-sensitive):
  • property_type values are now lower case, eg residential.

​

✨ New:

• first_seen_at column in the contractor table.
• employees table, linked to contractors table:
  • Includes a wide range of firmographic data for individual employees.
  • See “Schema Updates” below for the full list of columns.

​

🚀 Upgrades:

• additional columns in the residents table. See “Schema Updates” below for the full list.

​

💥 Breaking:

• Casing standardizations (in addition to the changes outlined in the API Section previously).
  • property_owner_type values are now snake_case, eg individual or company_owned.
  • owner_name, owner_street, owner_city values are now upper case, eg JANE DOE, MAIN ST, or AGAWAM.
  • applicant_name, applicant_street, applicant_city values are now upper case, just like owner_$ above.
• (PLANNED) Replacement of permits_ids with address_id in the residents table (please update your pipelines that use this foreign key value.)
  • This change will go into effect January 2025.