Changelog
Public version of FileFeeds embedding SDK in JS + React
Importer SDK versions 0.6.0 and 0.7.0
A new version of our SDKs for the Importer (0.6.0) is now available for all 4 supported frameworks: JS, React, Vue and Angular.
- Added compatibility with OneSchema FileFeeds packages.
- Exported TypeScript types in Importer React package.
- Dropped TemplateGroups feature.
- Added tslib as direct dependency.
Importer React SDK 0.7.0 update: inline
prop is defaulted to true
When upgrading to this version, if you still need the non-inline behavior, you could set inline={false}
explicitly. Or, you could switch to the inline
rendering (which is more native to React environment) and adjust your style attribute and CSS rules to achieve the desired UI presentation.
Webhook Logs for all webhooks
OneSchema now has logs for all types of supported webhooks:
- Importer Event webhooks
- Importer Metadata webhooks
- FileFeed Event webhooks
FileFeeds Mapping + Transforms experience updates
- Pipelines are now built into FileFeeds and called Transforms
- FileFeeds now respect pushed templates (relying on the latest template version pushed to the Production environment)
- Change history: you can now see the history of saved transforms per FileFeed
- New bulk action & pop-up: “Missing required columns”, clicking on “Add all missing columns” forces all required template columns onto the FileFeed
- “Save transforms” button moved to the top right corner of Transforms
- When uploading a new sample file to a FileFeed, the column mapping modal auto-opens
- Sample file columns mapped to template columns are now highlighted by a green “MAPPED” badge
- Other minor UI/UX updates
XML Pre-Parsing Hook
Pipelines JSON to CSV Pre-Parsing Hook
The new “JSON to CSV” pre-parsing hook for Pipelines allows you to extract fields from JSON: it creates a new column for each key in the JSON object and fills the cells with the respective values. The feature also supports array de-nesting: it duplicates the row for every array item.
Custom JSON Metadata for FileFeeds
Users can now set custom JSON metadata on FileFeeds via the settings page or via API. The metadata will be attached to validation webhook payloads, import webhook payloads, and to JSON file exports as file_feed_metadata
. There is a 2000 character limit and this metadata will not be attached to CSV exports.
FileFeed Import Webhooks Destination
Users can now use an import webhook as a file feed destination! Note that import webhooks for the Importer and FileFeeds product are separate - you may find tabs on the webhook page in your dashboard to delineate between the two.
Manual Fix for FileFeeds
Users can now manually fix file feed errors: open a FileFeed run that has “Error rows remaining”, click “Manually fix errors”, and it will open the familiar experience that OneSchema Importer users know as “Review & finalize.”
UPC, EAN, IMEI validation types
We're excited to announce the addition of 3 new validation types:
- UPC (Universal Product Code): is a barcode symbology that is used worldwide for tracking trade items in stores.
- EAN (International Article Number or European Article Number): A superset of UPC.
- IMEI (International Mobile Equipment Identity): a numeric identifier, usually unique, for mobile phones.
Saving Deleted Columns in Historical Mappings
If an end-user clicks on the “bin” icon next to a column on the Mapping step (which triggers the “This column will not be imported” message) - this action will now be saved in historical mappings, and reapplied for future uploads with the same column names.
Fixed Width File to CSV Converter
This newpreparsing hook allows you to accept fixed width files (e.g. TXT, DAT) and convert them to CSV.
Choose an Export Format for FileFeeds
Users can now specify a desired export format for their FileFeed output via the settings pane. Currently, CSV and JSON are supported.
Advanced Excel Preparsing Hook
We've launched a new preparsing hook that can be used for advanced Excel parsing. One example use case for this is the ability to ignore Excel's scientific notation formatting.
File Filter for FileFeeds
Users can specify a regex for filtering which files are processed from their FileFeed source. This is applied at the FileFeed level and can be changed in FileFeed settings or upon instantiation of a new FileFeed. Files in the source directory that don't match the regex will be ignored.
Multi-Mappable Columns
Customers can now specify a template column as multi-mappable, which allows your users to map multiple uploaded columns to the same template column. This enables better functionality for use cases such as aggregating multiple columns into a single column, and managing validation of multiple columns of the same type.
Custom Downloadable Excel Template
We're excited to just expand the capabilities of theExcel template file that your users can download during the import process! This new feature enables you to upload a custom file in place of our default generated one. You can find the configuration for this in the Settings tab of each template. Please note that the customization must be enabled for the download button to appear in the importer.
Validation Hooks page: a new Columns column
We've launched a new column to the table of Template Validation Hooks, listing the Template Columns attached to each hook. This is to help template owners manage and verify the hooks in use.
Support for 50M row files
We've increased the efficiency of our Rust data infrastructure, allowing support for uploading even larger files than before (up to 50M rows). Note: performance will be affected if code hooks are used, depending on the complexity of the code hook.
Access Control
Access Control is a feature that is now available for all Enterprise customers! With Access Control, three user roles are available:
- Admin: This user can manage your team’s access to OneSchema in addition to the Developer role’s permissions.
- Developer: This user will have access to the full OneSchema product experience. This includes templates, environments, API keys, webhooks, customizations, and analytics.
- Analyst: This user is able to view imports in Import Activity
Locked Columns
The new Locked Columns feature allows template owners to lock the value of specific columns, which is useful for fields where you don't want users to be able to edit certain data types. Once locked, the cells in these columns will be read-only, highlighted with a darker gray background, and have a popover indicating their lock status. If there are any errors/warnings for these cells, the error/warning message is displayed as before. Autofixing and accepting suggestions is enabled, however, manual edits are not.
New sheet_operation_performed event
With our new sheet_operation_performed event, you'll now be able to receive event webhooks for all events that happen on Pane 4. This gives you greater insights into the end user behavior which helps both with finding ways to improve your product, as well as building in-depth auditing capabilities.
Add a row during upload
We've launched a new customization that allows your users to add a row to uploaded files in importer! The added row will always appear at the end of the file.
Code Hooks API
We've launched additional endpoints for creating code hooks via the API to allow you to easily manage your code hooks locally.
Template Environments Update and Unpushed Changes
We've launched a new experience on template pages around environments that will enable admins to better manage making and pushing changes to templates. Admins will now see UI on the All Templates page that indicates which environments a template was last pushed to, and UI that indicates the number of changes made to a template that have not been pushed.
Frequent Error Types
Frequent Error Types has now been added to the Importer Analytics! This lets you see which columns your users are experiencing the most validation errors on the Review & Finalize pane.
Importer Analytics
Importer Analytics unlocks powerful information about how your customers are interacting with your importer. From here, you can see data like conversion percentages, time spent per pane, and frequent errors your customers are encountering.
Event Webhooks
For complex use cases where diving even further into analytics is necessary, Event Webhooks allows you to see data around the following events:
- Embed Initialized
- Embed Resumed
- Embed Closed
- File Uploaded
- Header Selected
- Columns Mapped
- Import Submitted
Import activity search filter
The new search filter on Import Activity provides a more seamless experience by enabling you to filter import activity across template_key, original file name, and user_id.
New Warnings Experience
We've made updates to our pane 4 experience to create a better experience when the uploaded file has warnings. With this new experience, when your user clicks the "Rows with issues" tab, they'll see a dropdown which allows them to filter on only errors OR only warnings. They'll also be able to filter on warnings for a specific column with the new warning pills, and the error sidebar now separates warnings from errors.
Additional error resolution actions
Our newly launched column level error resolution actions in pane 4 allow your users will to access the predefined actions via error popover and via column header dropdowns. This gives them more flexibility and options when it comes to bulk fixing their data, enabling an even easier data cleaning experience!
Webhook logs popover update
The log details popover for Validation Webhooks now contains "rows sent" and "errors returned". This will aid you in more easily debugging validation webhook issues.
All templates page redesign
We've just launched a redesign to the all templates page that allows users to search and sort their templates by various criteria (eg. template key, label, created time) and filter templates by whether they were created from the API. Our goal is to continue making it easier and much faster for you to find a template they are looking for, especially for organizations with a large amount of dynamic templates.
Import max row limit
We've added a customization that allows customers to add a maximum row limit that can be imported, and they can also add an optional custom header and description message for this error. When this customization is active and there are more rows than the specified limit, we will display an error in the importer and block the user from continuing with the import.
Education widget
We've added a customization option that allows you to add education widgets to your importer! This enables you specify different widgets for each of the 4 Importer panes, have it default open or closed, and supports markdown.
Environment scopes for API keys
We're excited to roll out environment scopes for API keys, which enables those keys to only access embeds in the prescribed environment. This helps customers uplevel the security of their organization by ensuring that their developers' local and staging environments should not have access to customer PII.
Accept all suggestions
The cell popover for errors and warnings with suggestions now has a new button, "Accept for X cells". Clicking this button will accept all suggestions for cells in the column that have exactly 1 suggestion, making it even easier for your users to clean their files. The button will appear for users if there are at least 2 cells in the column which have exactly 1 suggestion.
Picklist Descriptions
With Picklist Descriptions, you can now use template overrides to set descriptions for each individual picklist option. These will be displayed in each dropdown of the OneSchema importer:
- Picklist mapping
- Picklist cell editing
- Picklist error popovers to replace all
Picklist Truncation and Dropdown Width Adjustments
These 3 improvements to picklists help customers whose use cases require long picklist values to be inputted:
- Picklist pills will now truncate properly with a tooltip on hover to show the full value
- The base width of the dropdown has just naturally been increased 20% across the board
- The width of the dropdown will now actually increase if the column width increases
Template Columns Education
This new customization option under the "User Education" tab enables an informational sidebar on the mapping pane for your users. This will display a green check next to columns that have been mapped, and update dynamically as users select their mappings.
Sample data file generator
Users now have the ability to generate a sample test file for a template, eliminating the need for a file on hand to experience the importer. The button to generate a sample file lives can be found in the sandbox preview below the importer. The generated data will mostly conform with the template column options, but the data may not be 100% valid according to the template.
Accounting format for Number Autofixer
We've launched an update to our number autofixer, which can now handle negative numbers in account format. You can utilize this to automatically correct entries formatted as (100), and automatically change them to -100.
Alternative Picklist Names
Our new Alternative Picklist Names feature enables you to specify one or more "alternative_names" for each picklist value via template overrides. If any of the alternative names appear in the uploaded file, they will automatically be mapped to the picklist value.
EU Number Format
The number data type now has a new selection for the format, either be "US" (the default), or "EU". The EU number data type uses "," as the decimal separator and "." as the thousand separator.
External API keys update
The external API keys page has been updated so every user can see all external API keys in their organization. With these changes, a user can create multiple keys, set a label, and see when an API key was last used. In addition, deleting a user also now does not delete any API key(s) created by that user.
Fullview integration
OneSchema now supports Fullview as a screen recording provider, allowing customers to analyzing recordings of their customers using OneSchema and optimize the experience.
Required Column Groups
Customers can now use the "Required Column Groups" feature to set that at least one column in a group must be mapped.
New Boolean data type
Our new Boolean data type feature allows users to define true and false values more easily.
Customizable Picklist Colors
You can now set specific colors for each picklist option! A list of 10 default options are provided, but exact HEX values can be used as desired.
Environment variables in webhook keys
Customers can now set environment variables in webhook keys. This helps customers who want to keep their secret keys isolated across different environments.
OneSchema Pipelines
OneSchema Pipelines lets non-technical team members (account managers, operations) setup recurring CSV integrations with your customers’ complex data feeds without engineers getting involved. Map, transform, and ingest CSV files via SFTP, API, or email without needing to write a single line of code.
Import template as JSON
Define validation hooks in template overrides
Customizations and Templates determine the behavior of OneSchema Importers. Using overrides can allow specific behavior for individual importing sessions. Customers now have the ability to define validation hooks in template overrides.
Historical Matching improvements
We've launched a feature to split Historical Matching into User-specific and Org-level matching. Previously, this only worked on the org-level. This helps customers whose users are differentiated enough that they do not want to have cross-user mappings be saved.
Environment Variables in Importer Webhooks
Environments General Availability
Environments allow you to safely push and validate changes to templates in stages to align with the environments in your deploy process.
OneSchema automatically sets up environments for Production, Staging, and Development. See our guide on custom environments to set up custom environments.
Per-Customer Overrides
We’ve made improvements to our Template Overrides functionality by adding the ability to add and remove columns from the overrides.
Code Hooks improvements
We’ve made improvements to our custom code transformations and validations, also known as OneSchema Code Hooks. These Code Hook upgrades aim to greatly enhance both how your team builds custom functions that support your company’s unique business logic, as well as how your end-users experience the OneSchema Importer.
Transpose file
A new post-upload hook now exists for transposing (swapping the rows and columns) of a file.
Multi-cell copy and paste
To better support end users with bulk error resolution, we’ve improved our copy and paste functionality on the Review & Finalize pane to enable multiple cell copying and pasting.
Review & Finalize pane UI improvements
We’ve added UI for end users to immediately filter for all rows, only rows with errors, or only clean rows.
String Replacements customization
String Replacements allow you to change the headers and buttons of every step of the importer. This can be helpful if there’s an existing string inside of the importer flow that you feel could be edited to better fit your existing brand (or use case).
Row deletion customization
For some customers, an end user deleting a row of data prior to import can break their entire flow. We’ve added a customization option that allows you granular control over how and if the end user should be able to delete rows from the Review & Finalize pane.
Specify expected date formats for template columns
Admins can now specify the particular date format that’s expected for template columns instead of relying on OneSchema’s date detection (which defaults to a month, day, year interpretation). This impacts scenarios where a file’s uploaded date column contains a majority of ambiguous dates, and the format is not MM/DD/YY (e.g. 01/02/2022 can be either January 2nd, 2022 or February 1st, 2022).
Custom column support in validation webhooks
Custom columns can now be supported inside of validation webhooks. Review our updated docs to understand the different use cases, how to set up custom columns, and example JSON POST/responses.
Pre-built Transforms and Advanced Validations
Markdown support in customizations
To provide further customization of our Upload pane, the optional message box can now be customized using Markdown. Our customers can insert URLs to additional data import resources for the end user.
JSON object data type validator
Multi-suggestion support for validation web hooks
Customers can now choose to return a list of suggestions from a validation webhook. End users will be prompted to select one of the suggestions as a part of the error resolution process.
Currency symbol validation support
OneSchema’s money data type validation and autofix feature now supports the Pound (£), the Euro (€), and the Yen (¥) currency symbols.
Better error messages for numbers and currency
The error messaging for the Number data type and Money data type has been improved so end users can better understand why the value is invalid and how to fix it.
Import metadata via webhook
Rather than sending batched rows of data to a customer webhook, sheet metadata can now be sent via webhook. This is significantly faster and also enables you to directly use the metadata to retrieve data via external API (e.g. GET rows for paginated JSON or POST sheets for S3 URL download of CSV/excel/JSON).
Headless validation
Since we’ve started building OneSchema, one of the deepest pain points we’ve seen in our customer base is building and maintaining recurring spreadsheet import workflows. Expensive engineering teams have to write brittle scripts to ingest the data, and spend countless engineering months maintaining validations to make sure all customer data is being imported correctly.
In the case of a failure, it’s extremely painful for the technical team to track down the root cause of the failure, and all the while the customer is frustrated because their data isn’t up-to-date in your product. OneSchema Headless empowers engineering teams to leverage OneSchema’s mapping, validation, and error resolution capabilities to painlessly build custom integrations with a wide variety of data sources, like SFTP/S3 file drops, CSV email attachments, messy API data, and more.
Webhook logs
Users can now see logs of their importer and validation webhooks (up to the past 14 days). This should enable users to better debug their webhooks when they run into errors, and enable your teams to monitor webhook activity.
Direct to S3 file transfer
Customers can now instruct OneSchema to send processed files directly to their own presigned URL.
CA data residency
AU data residency
In-memory code hooks
Because our customers span a variety of industries, we often see workflows that come with unique business logic built into their specific datasets.
To make OneSchema even more flexible for these custom use cases, we’ve created OneSchema Code Hooks. Code Hooks empower your engineering team to generate tailored functions to reformat, transform, and/or validate data specific to your business's needs inside of the importer flow.
Historical picklist mapping
Similar to historical column mapping, historical picklist option mapping remembers mappings and pre-fills the values for subsequent uploads. This behavior is active if the “Historical match” customization in the “Map columns” section is set.
Resume sessions
Sessions can now be resumed in case a user's browser crashes or they do not finish an import. To enable this feature, update your SDK to the latest version(0.3.3) and pass in the param saveSession as true. Then sessions using the same userJwt and template key on the same browser will be prompted on resume as seen in the screenshot below.
New template configurations
OneSchema templates just got even more powerful.
- With the alternative mappings feature, you can tell OneSchema which mappings you’re expecting to see so that your user won’t have to map them manually.
- Use fill default values to automatically fill in empty cells instead of leaving them blank
- Check out the flexible options on our data type validations that let you validate everything from digits after the decimal to excluding special characters.
We've also redesigned the template column creation modal to better organize all the different data validations options and to make template creation easier for you.
Self hosting (Enterprise Feature)
You can now host OneSchema inside of your AWS or Azure cloud environment. The self-hosted model provides full isolation of data in your own cloud, and is best for customers who are handling government data (GovCloud), or other extremely sensitive data with unique contractual / compliance requirements. OneSchema will deploy updates and maintenance to the platform through an IAM user.
Customizations Dashboard
A dashboard with granular UI customizations is now available for you to make the OneSchema Importer look and feel like your existing product UI! We’ve also updated our default icon set and illustrations to be more brand-neutral, to help OneSchema’s Importer fit better in any product, even without customizations.
A dashboard with granular UI customizations is now available for you to make the OneSchema Importer look and feel like your existing product UI! We’ve also updated our default icon set and illustrations to be more brand-neutral, to help OneSchema’s Importer fit better in any product, even without customizations.
With our Branding Suite, you can now access 20+ customization options, including:
- Primary, secondary, and background colors
- Buttons
- Success, Warning, and Error states
- Custom illustrations
- Font family and font colors
Watch this 3 minute step-by-step demo of the OneSchema Branding Suite that demonstrates how to fully customize your importer's color, font, and more.
Accept clean rows only
A new modal has been added to the end of the Review & Finalize pane that allows you to accept only rows without errors. For use cases where you’d like to get as much data in as possible but ingesting all of the user’s data isn’t critical, this is a great option to increase your import conversion rates.
The new modal addition:
- Allows your user to submit clean rows without having to fix every single error in their data
- Gives end users the opportunity to export the remaining errors so they may fix their data in Excel and re-upload it into your system.
Note: This modal will only appear if there are remaining error rows in your end user’s upload when they attempt to import.
OneSchema will still send the final JSON export with clean rows as “records” and error rows as “error_records.” This way, even if your user never returned to clean the rows with the remaining errors, you still have a record of the data not imported.
Export to S3
OneSchema webhooks let you send data to your ingestion endpoint but can be slow for very large files.
Now, OneSchema can put your exported file into an S3 bucket, where you can retrieve it as JSON or CSV. This makes exporting large files far more performant.
To enable this new feature, a new OneSchemaConfig has been added: skipExportData. This changes the behavior of the success event (onSuccess callback in React) when not including a webhookKey. Instead of sending all of the data from the list, we will only send the sheet_id. You can then use API v1 to download a file or use our Paginated JSON Get end point.
We recommend this config option for customers who anticipate their end users will upload large files but don’t want them to wait for the file to finish processing before proceeding back to their workflow.
You can learn more about the config in our docs.
Public version of FileFeeds embedding SDK in JS + React
Importer SDK versions 0.6.0 and 0.7.0
A new version of our SDKs for the Importer (0.6.0) is now available for all 4 supported frameworks: JS, React, Vue and Angular.
- Added compatibility with OneSchema FileFeeds packages.
- Exported TypeScript types in Importer React package.
- Dropped TemplateGroups feature.
- Added tslib as direct dependency.
Importer React SDK 0.7.0 update: inline
prop is defaulted to true
When upgrading to this version, if you still need the non-inline behavior, you could set inline={false}
explicitly. Or, you could switch to the inline
rendering (which is more native to React environment) and adjust your style attribute and CSS rules to achieve the desired UI presentation.
Webhook Logs for all webhooks
OneSchema now has logs for all types of supported webhooks:
- Importer Event webhooks
- Importer Metadata webhooks
- FileFeed Event webhooks
XML Pre-Parsing Hook
Pipelines JSON to CSV Pre-Parsing Hook
The new “JSON to CSV” pre-parsing hook for Pipelines allows you to extract fields from JSON: it creates a new column for each key in the JSON object and fills the cells with the respective values. The feature also supports array de-nesting: it duplicates the row for every array item.
Custom JSON Metadata for FileFeeds
Users can now set custom JSON metadata on FileFeeds via the settings page or via API. The metadata will be attached to validation webhook payloads, import webhook payloads, and to JSON file exports as file_feed_metadata
. There is a 2000 character limit and this metadata will not be attached to CSV exports.
FileFeed Import Webhooks Destination
Users can now use an import webhook as a file feed destination! Note that import webhooks for the Importer and FileFeeds product are separate - you may find tabs on the webhook page in your dashboard to delineate between the two.
Manual Fix for FileFeeds
Users can now manually fix file feed errors: open a FileFeed run that has “Error rows remaining”, click “Manually fix errors”, and it will open the familiar experience that OneSchema Importer users know as “Review & finalize.”
UPC, EAN, IMEI validation types
We're excited to announce the addition of 3 new validation types:
- UPC (Universal Product Code): is a barcode symbology that is used worldwide for tracking trade items in stores.
- EAN (International Article Number or European Article Number): A superset of UPC.
- IMEI (International Mobile Equipment Identity): a numeric identifier, usually unique, for mobile phones.
Saving Deleted Columns in Historical Mappings
If an end-user clicks on the “bin” icon next to a column on the Mapping step (which triggers the “This column will not be imported” message) - this action will now be saved in historical mappings, and reapplied for future uploads with the same column names.
Fixed Width File to CSV Converter
This newpreparsing hook allows you to accept fixed width files (e.g. TXT, DAT) and convert them to CSV.
Choose an Export Format for FileFeeds
Users can now specify a desired export format for their FileFeed output via the settings pane. Currently, CSV and JSON are supported.
Advanced Excel Preparsing Hook
We've launched a new preparsing hook that can be used for advanced Excel parsing. One example use case for this is the ability to ignore Excel's scientific notation formatting.
File Filter for FileFeeds
Users can specify a regex for filtering which files are processed from their FileFeed source. This is applied at the FileFeed level and can be changed in FileFeed settings or upon instantiation of a new FileFeed. Files in the source directory that don't match the regex will be ignored.
Multi-Mappable Columns
Customers can now specify a template column as multi-mappable, which allows your users to map multiple uploaded columns to the same template column. This enables better functionality for use cases such as aggregating multiple columns into a single column, and managing validation of multiple columns of the same type.
Custom Downloadable Excel Template
We're excited to just expand the capabilities of theExcel template file that your users can download during the import process! This new feature enables you to upload a custom file in place of our default generated one. You can find the configuration for this in the Settings tab of each template. Please note that the customization must be enabled for the download button to appear in the importer.
Validation Hooks page: a new Columns column
We've launched a new column to the table of Template Validation Hooks, listing the Template Columns attached to each hook. This is to help template owners manage and verify the hooks in use.
Access Control
Access Control is a feature that is now available for all Enterprise customers! With Access Control, three user roles are available:
- Admin: This user can manage your team’s access to OneSchema in addition to the Developer role’s permissions.
- Developer: This user will have access to the full OneSchema product experience. This includes templates, environments, API keys, webhooks, customizations, and analytics.
- Analyst: This user is able to view imports in Import Activity
Locked Columns
The new Locked Columns feature allows template owners to lock the value of specific columns, which is useful for fields where you don't want users to be able to edit certain data types. Once locked, the cells in these columns will be read-only, highlighted with a darker gray background, and have a popover indicating their lock status. If there are any errors/warnings for these cells, the error/warning message is displayed as before. Autofixing and accepting suggestions is enabled, however, manual edits are not.
New sheet_operation_performed event
With our new sheet_operation_performed event, you'll now be able to receive event webhooks for all events that happen on Pane 4. This gives you greater insights into the end user behavior which helps both with finding ways to improve your product, as well as building in-depth auditing capabilities.
Add a row during upload
We've launched a new customization that allows your users to add a row to uploaded files in importer! The added row will always appear at the end of the file.
Template Environments Update and Unpushed Changes
We've launched a new experience on template pages around environments that will enable admins to better manage making and pushing changes to templates. Admins will now see UI on the All Templates page that indicates which environments a template was last pushed to, and UI that indicates the number of changes made to a template that have not been pushed.
Frequent Error Types
Frequent Error Types has now been added to the Importer Analytics! This lets you see which columns your users are experiencing the most validation errors on the Review & Finalize pane.
Importer Analytics
Importer Analytics unlocks powerful information about how your customers are interacting with your importer. From here, you can see data like conversion percentages, time spent per pane, and frequent errors your customers are encountering.
Event Webhooks
For complex use cases where diving even further into analytics is necessary, Event Webhooks allows you to see data around the following events:
- Embed Initialized
- Embed Resumed
- Embed Closed
- File Uploaded
- Header Selected
- Columns Mapped
- Import Submitted
Import activity search filter
The new search filter on Import Activity provides a more seamless experience by enabling you to filter import activity across template_key, original file name, and user_id.
New Warnings Experience
We've made updates to our pane 4 experience to create a better experience when the uploaded file has warnings. With this new experience, when your user clicks the "Rows with issues" tab, they'll see a dropdown which allows them to filter on only errors OR only warnings. They'll also be able to filter on warnings for a specific column with the new warning pills, and the error sidebar now separates warnings from errors.
Additional error resolution actions
Our newly launched column level error resolution actions in pane 4 allow your users will to access the predefined actions via error popover and via column header dropdowns. This gives them more flexibility and options when it comes to bulk fixing their data, enabling an even easier data cleaning experience!
Import max row limit
We've added a customization that allows customers to add a maximum row limit that can be imported, and they can also add an optional custom header and description message for this error. When this customization is active and there are more rows than the specified limit, we will display an error in the importer and block the user from continuing with the import.
Education widget
We've added a customization option that allows you to add education widgets to your importer! This enables you specify different widgets for each of the 4 Importer panes, have it default open or closed, and supports markdown.
Accept all suggestions
The cell popover for errors and warnings with suggestions now has a new button, "Accept for X cells". Clicking this button will accept all suggestions for cells in the column that have exactly 1 suggestion, making it even easier for your users to clean their files. The button will appear for users if there are at least 2 cells in the column which have exactly 1 suggestion.
Picklist Descriptions
With Picklist Descriptions, you can now use template overrides to set descriptions for each individual picklist option. These will be displayed in each dropdown of the OneSchema importer:
- Picklist mapping
- Picklist cell editing
- Picklist error popovers to replace all
Template Columns Education
This new customization option under the "User Education" tab enables an informational sidebar on the mapping pane for your users. This will display a green check next to columns that have been mapped, and update dynamically as users select their mappings.
Sample data file generator
Users now have the ability to generate a sample test file for a template, eliminating the need for a file on hand to experience the importer. The button to generate a sample file lives can be found in the sandbox preview below the importer. The generated data will mostly conform with the template column options, but the data may not be 100% valid according to the template.
Alternative Picklist Names
Our new Alternative Picklist Names feature enables you to specify one or more "alternative_names" for each picklist value via template overrides. If any of the alternative names appear in the uploaded file, they will automatically be mapped to the picklist value.
EU Number Format
The number data type now has a new selection for the format, either be "US" (the default), or "EU". The EU number data type uses "," as the decimal separator and "." as the thousand separator.
Fullview integration
OneSchema now supports Fullview as a screen recording provider, allowing customers to analyzing recordings of their customers using OneSchema and optimize the experience.
Required Column Groups
Customers can now use the "Required Column Groups" feature to set that at least one column in a group must be mapped.
New Boolean data type
Our new Boolean data type feature allows users to define true and false values more easily.
Customizable Picklist Colors
You can now set specific colors for each picklist option! A list of 10 default options are provided, but exact HEX values can be used as desired.
OneSchema Pipelines
OneSchema Pipelines lets non-technical team members (account managers, operations) setup recurring CSV integrations with your customers’ complex data feeds without engineers getting involved. Map, transform, and ingest CSV files via SFTP, API, or email without needing to write a single line of code.
Import template as JSON
Environments General Availability
Environments allow you to safely push and validate changes to templates in stages to align with the environments in your deploy process.
OneSchema automatically sets up environments for Production, Staging, and Development. See our guide on custom environments to set up custom environments.
Transpose file
A new post-upload hook now exists for transposing (swapping the rows and columns) of a file.
Multi-cell copy and paste
To better support end users with bulk error resolution, we’ve improved our copy and paste functionality on the Review & Finalize pane to enable multiple cell copying and pasting.
String Replacements customization
String Replacements allow you to change the headers and buttons of every step of the importer. This can be helpful if there’s an existing string inside of the importer flow that you feel could be edited to better fit your existing brand (or use case).
Row deletion customization
For some customers, an end user deleting a row of data prior to import can break their entire flow. We’ve added a customization option that allows you granular control over how and if the end user should be able to delete rows from the Review & Finalize pane.
Specify expected date formats for template columns
Admins can now specify the particular date format that’s expected for template columns instead of relying on OneSchema’s date detection (which defaults to a month, day, year interpretation). This impacts scenarios where a file’s uploaded date column contains a majority of ambiguous dates, and the format is not MM/DD/YY (e.g. 01/02/2022 can be either January 2nd, 2022 or February 1st, 2022).
Pre-built Transforms and Advanced Validations
JSON object data type validator
Currency symbol validation support
OneSchema’s money data type validation and autofix feature now supports the Pound (£), the Euro (€), and the Yen (¥) currency symbols.
Import metadata via webhook
Rather than sending batched rows of data to a customer webhook, sheet metadata can now be sent via webhook. This is significantly faster and also enables you to directly use the metadata to retrieve data via external API (e.g. GET rows for paginated JSON or POST sheets for S3 URL download of CSV/excel/JSON).
Headless validation
Since we’ve started building OneSchema, one of the deepest pain points we’ve seen in our customer base is building and maintaining recurring spreadsheet import workflows. Expensive engineering teams have to write brittle scripts to ingest the data, and spend countless engineering months maintaining validations to make sure all customer data is being imported correctly.
In the case of a failure, it’s extremely painful for the technical team to track down the root cause of the failure, and all the while the customer is frustrated because their data isn’t up-to-date in your product. OneSchema Headless empowers engineering teams to leverage OneSchema’s mapping, validation, and error resolution capabilities to painlessly build custom integrations with a wide variety of data sources, like SFTP/S3 file drops, CSV email attachments, messy API data, and more.
Webhook logs
Users can now see logs of their importer and validation webhooks (up to the past 14 days). This should enable users to better debug their webhooks when they run into errors, and enable your teams to monitor webhook activity.
Direct to S3 file transfer
Customers can now instruct OneSchema to send processed files directly to their own presigned URL.
In-memory code hooks
Because our customers span a variety of industries, we often see workflows that come with unique business logic built into their specific datasets.
To make OneSchema even more flexible for these custom use cases, we’ve created OneSchema Code Hooks. Code Hooks empower your engineering team to generate tailored functions to reformat, transform, and/or validate data specific to your business's needs inside of the importer flow.
Historical picklist mapping
Similar to historical column mapping, historical picklist option mapping remembers mappings and pre-fills the values for subsequent uploads. This behavior is active if the “Historical match” customization in the “Map columns” section is set.
Resume sessions
Sessions can now be resumed in case a user's browser crashes or they do not finish an import. To enable this feature, update your SDK to the latest version(0.3.3) and pass in the param saveSession as true. Then sessions using the same userJwt and template key on the same browser will be prompted on resume as seen in the screenshot below.
Customizations Dashboard
A dashboard with granular UI customizations is now available for you to make the OneSchema Importer look and feel like your existing product UI! We’ve also updated our default icon set and illustrations to be more brand-neutral, to help OneSchema’s Importer fit better in any product, even without customizations.
A dashboard with granular UI customizations is now available for you to make the OneSchema Importer look and feel like your existing product UI! We’ve also updated our default icon set and illustrations to be more brand-neutral, to help OneSchema’s Importer fit better in any product, even without customizations.
With our Branding Suite, you can now access 20+ customization options, including:
- Primary, secondary, and background colors
- Buttons
- Success, Warning, and Error states
- Custom illustrations
- Font family and font colors
Watch this 3 minute step-by-step demo of the OneSchema Branding Suite that demonstrates how to fully customize your importer's color, font, and more.
Accept clean rows only
A new modal has been added to the end of the Review & Finalize pane that allows you to accept only rows without errors. For use cases where you’d like to get as much data in as possible but ingesting all of the user’s data isn’t critical, this is a great option to increase your import conversion rates.
The new modal addition:
- Allows your user to submit clean rows without having to fix every single error in their data
- Gives end users the opportunity to export the remaining errors so they may fix their data in Excel and re-upload it into your system.
Note: This modal will only appear if there are remaining error rows in your end user’s upload when they attempt to import.
OneSchema will still send the final JSON export with clean rows as “records” and error rows as “error_records.” This way, even if your user never returned to clean the rows with the remaining errors, you still have a record of the data not imported.
Export to S3
OneSchema webhooks let you send data to your ingestion endpoint but can be slow for very large files.
Now, OneSchema can put your exported file into an S3 bucket, where you can retrieve it as JSON or CSV. This makes exporting large files far more performant.
To enable this new feature, a new OneSchemaConfig has been added: skipExportData. This changes the behavior of the success event (onSuccess callback in React) when not including a webhookKey. Instead of sending all of the data from the list, we will only send the sheet_id. You can then use API v1 to download a file or use our Paginated JSON Get end point.
We recommend this config option for customers who anticipate their end users will upload large files but don’t want them to wait for the file to finish processing before proceeding back to their workflow.
You can learn more about the config in our docs.
Import via API
We understand every company's file ingestion process is unique. That’s why a focus for OneSchema has always been enabling companies to optimize OneSchema’s import workflow for their specific use case. To further expand on the customizability of OneSchema, we’re introducing Import via API.
For workflows that ideally involve a blend of managing some portions of the import process internally while still using OneSchema to handle the rest, initializing the importer via our API provides a great solution.
Accept partial imports
A new modal has been added to the end of the Review & Finalize pane that allows you to accept only rows without errors. For use cases where you’d like to get as much data in as possible but ingesting all of the user’s data isn’t critical, this is a great option to increase your import conversion rates.
The new modal addition:
- Allows your user to submit clean rows without having to fix every single error in their data
- Gives end users the opportunity to export the remaining errors so they may fix their data in Excel and re-upload it into your system.
Note: This modal will only appear if there are remaining error rows in your end user’s upload when they attempt to import.
OneSchema will still send the final JSON export with clean rows as “records” and error rows as “error_records.” This way, even if your user never returned to clean the rows with the remaining errors, you still have a record of the data not imported.
Skip mapping pane
The importer will now skip the mapping step if all template columns are matched (exact match) in the list (and header row selection was also skipped).
Data Template summary
You can now customize the first pane of your CSV import experience to include a summary of the data you’d like your customer to import. You can include your required and optional fields and descriptions of the data the customer should upload.
We added the following optional customizations to empower you to create the ideal experience for your end users on the first pane:
- The header and subheader copy within the Uploader component can now be fully customized
- Show the required columns and their descriptions in a new InfoSideBar component
- Show the required and optional columns along with their descriptions in a new InfoSideBar component
You can learn more about how to enable these customizations in our docs.
Localization (21 languages)
OneSchema now supports the ability to fully translate the UI into the language of your choosing. Internationalization is currently an enterprise-tier feature. Please reach out to us to discuss your team's i18n needs and desired language support. Supported languages include: English, Spanish, Portuguese, French, German, Swedish, Chinese (Simplified), Czech, Danish, Dutch, Finnish, Hungarian, Italian, Japanese, Korean, Norwegian, Polish, Portuguese, Romanian, Russian, Arabic,
Learn more about the configuration option in our docs.
Upload pane customizations (v1)
Multi-picklist error resolution
We’ve improved our picklist data type error resolution to support multi-value picklists. Your users can now resolve any delimited picklist columns with ease in the same way they would for single value picklist columns.
Skip header row selection pane
To streamline the import process, users who upload a file with a header row will now skip the Header Row Selection pane and be taken directly to the Map Column pane. Header rows are automatically identified based on a number of criteria. To learn more about how we identify the header row, visit our changelog here.
Template sandbox experience
To enhance the template creation process as well as the implementation experience, we’ve released a sandbox of the Importer. This will allow you to see the Importer in action without any additional effort and quickly test new or existing templates with real files.
Picklist mapping
Your users will now be prompted to map their picklist values to the set of values that you specify within a template. Picklist mapping will help eliminate any unrecognized, undesired, or incorrect values from being in the imported data.
Intelligent column mapping
We’ve improved our column matching algorithm so your users’ uploaded columns will be automatically mapped to template columns during the Map Column pane more often. In addition to mapping columns that have exact column names, OneSchema will also remember historical mappings that your users may have successfully imported in the past. You can learn more about how our mapping intelligence works in our changelog.
Angular SDK
Angular SDK is now available for users who would prefer to build the OneSchema Importer using Angular.
Branding v1
You’re now able to customize the appearance of the OneSchema Importer through our developer dashboard! You may customize the primary color of the Importer to better match your brand colors and import a custom font via URL (e.g. Google Fonts or Adobe TypeKit). You will also be able to hide the OneSchema logo as long as you have committed to our annual subscription pricing.
We’re actively working on bringing you more granular customization options to better fit in with your product styles — if you have any particular requests, please let us know.
Custom columns
From our new Template Settings, you can now allow your users to upload and import custom columns from the Importer.
Your users can map any of their uploaded columns as a custom column. Custom columns come with default string validations but can be customized with additional validations from the Template Settings page.
Column validation hooks
External validation hooks now also support column hooks in Rust. Below are the differences between the existing external row hooks and the new external column hooks:
Configuration differences: If the hook type is `column` the batch size can’t be configured since we always send all rows for column hooks. As you can see in the screenshot we also show a warning banner since a slow column hook could result in a bad user experience. The reason for this lies in the behavioral differences.
Behavioral differences: When a single value is updated by the user we send only send the edited row for row hooks. For column hooks, we send the whole affected column which can be slower depending on the logic implemented by our admins.
Gating: The gate `ColumnHooks` has to be active in order to create and use column hooks. If the batch size doesn’t show up the `InMemoryListStorage` gate is not active.
Embedded importer launch
The launch of our ready-made CSV importer for developers which automatically corrects customer data. (Also: edge case handling, simple UI, bulk editing).
Dynamic templates
Create custom templates for each of your users. Generate templates via code and create an import experience tailored to your customer’s unique needs.
FileFeeds Mapping + Transforms experience updates
- Pipelines are now built into FileFeeds and called Transforms
- FileFeeds now respect pushed templates (relying on the latest template version pushed to the Production environment)
- Change history: you can now see the history of saved transforms per FileFeed
- New bulk action & pop-up: “Missing required columns”, clicking on “Add all missing columns” forces all required template columns onto the FileFeed
- “Save transforms” button moved to the top right corner of Transforms
- When uploading a new sample file to a FileFeed, the column mapping modal auto-opens
- Sample file columns mapped to template columns are now highlighted by a green “MAPPED” badge
- Other minor UI/UX updates
Support for 50M row files
We've increased the efficiency of our Rust data infrastructure, allowing support for uploading even larger files than before (up to 50M rows). Note: performance will be affected if code hooks are used, depending on the complexity of the code hook.
Code Hooks API
We've launched additional endpoints for creating code hooks via the API to allow you to easily manage your code hooks locally.
Webhook logs popover update
The log details popover for Validation Webhooks now contains "rows sent" and "errors returned". This will aid you in more easily debugging validation webhook issues.
All templates page redesign
We've just launched a redesign to the all templates page that allows users to search and sort their templates by various criteria (eg. template key, label, created time) and filter templates by whether they were created from the API. Our goal is to continue making it easier and much faster for you to find a template they are looking for, especially for organizations with a large amount of dynamic templates.
Picklist Truncation and Dropdown Width Adjustments
These 3 improvements to picklists help customers whose use cases require long picklist values to be inputted:
- Picklist pills will now truncate properly with a tooltip on hover to show the full value
- The base width of the dropdown has just naturally been increased 20% across the board
- The width of the dropdown will now actually increase if the column width increases
Accounting format for Number Autofixer
We've launched an update to our number autofixer, which can now handle negative numbers in account format. You can utilize this to automatically correct entries formatted as (100), and automatically change them to -100.
External API keys update
The external API keys page has been updated so every user can see all external API keys in their organization. With these changes, a user can create multiple keys, set a label, and see when an API key was last used. In addition, deleting a user also now does not delete any API key(s) created by that user.
Environment variables in webhook keys
Customers can now set environment variables in webhook keys. This helps customers who want to keep their secret keys isolated across different environments.
Define validation hooks in template overrides
Customizations and Templates determine the behavior of OneSchema Importers. Using overrides can allow specific behavior for individual importing sessions. Customers now have the ability to define validation hooks in template overrides.
Historical Matching improvements
We've launched a feature to split Historical Matching into User-specific and Org-level matching. Previously, this only worked on the org-level. This helps customers whose users are differentiated enough that they do not want to have cross-user mappings be saved.
Environment Variables in Importer Webhooks
Per-Customer Overrides
We’ve made improvements to our Template Overrides functionality by adding the ability to add and remove columns from the overrides.
Code Hooks improvements
We’ve made improvements to our custom code transformations and validations, also known as OneSchema Code Hooks. These Code Hook upgrades aim to greatly enhance both how your team builds custom functions that support your company’s unique business logic, as well as how your end-users experience the OneSchema Importer.
Review & Finalize pane UI improvements
We’ve added UI for end users to immediately filter for all rows, only rows with errors, or only clean rows.
Custom column support in validation webhooks
Custom columns can now be supported inside of validation webhooks. Review our updated docs to understand the different use cases, how to set up custom columns, and example JSON POST/responses.
Markdown support in customizations
To provide further customization of our Upload pane, the optional message box can now be customized using Markdown. Our customers can insert URLs to additional data import resources for the end user.
Multi-suggestion support for validation web hooks
Customers can now choose to return a list of suggestions from a validation webhook. End users will be prompted to select one of the suggestions as a part of the error resolution process.
Better error messages for numbers and currency
The error messaging for the Number data type and Money data type has been improved so end users can better understand why the value is invalid and how to fix it.
New template configurations
OneSchema templates just got even more powerful.
- With the alternative mappings feature, you can tell OneSchema which mappings you’re expecting to see so that your user won’t have to map them manually.
- Use fill default values to automatically fill in empty cells instead of leaving them blank
- Check out the flexible options on our data type validations that let you validate everything from digits after the decimal to excluding special characters.
We've also redesigned the template column creation modal to better organize all the different data validations options and to make template creation easier for you.
Guided error-fixing
We’re very excited to announce our most requested feature: guided error fixing. In our beta tests, offering helpful suggestions for resolving errors has driven substantial uplift in import conversion rates. These upgrades include:
- Error Fix Suggestions: OneSchema will offer suggestions to your customer to fix errors, contextualized to the type of error your customer is encountering.
- Navigate to column with errors: When clicked in the issues summary, OneSchema will now pull the column with issues directly into view.
Bulk deleting rows:
- Delete all rows with errors: OneSchema now contextually surfaces the option to delete all rows with errors, allowing customers to progress in their import workflow. Alternatively they can “Export to Excel” to get a summary of their errors.
- Delete selected rows: “Delete selected row(s)” button when selecting rows will be shown to users
We are also launching usability improvements based on customer feedback:
- Issues Sidebar: Issues summary sidebar will no longer obstruct the last few columns in the sheet. Instead, it will appear inside of the spreadsheet view.
- Filter to find errors: We’ve added a “show error” button in the issues summary that when clicked, brings the error front-and-center. It’s never been easier to find errors in your file.
Import activity
An activity feed of all successful, failed, and unsubmitted imports are now available to view via the Developer Dashboard. This will let your team review information about the files that are being imported by each of your users. To help your customer success teams easily provide troubleshooting support on failed imports, we’ve also added the option to download the original files uploaded by your users.
SDK updated to 0.2
We’ve updated the version of our Javascript and React SDKs to v0.2. Updating the version will require some small changes to initialization and passing in configuration options.
See the new documentation here:
Javascript: https://docs.oneschema.co/docs/javascript
Default fill columns
You can now enable template columns to automatically fill a value of your choice into the mapped column’s empty cells.
Note: This is only available to customers whose Importers are using our new Rust architecture. Please reach out to your account manager if this feature isn’t available in your Create a Template column modal.
Advanced Branding
You’re now able to customize the appearance of the OneSchema Importer through our developer dashboard! You may customize the primary color of the Importer to better match your brand colors and import a custom font via URL (e.g. Google Fonts or Adobe TypeKit). You will also be able to hide the OneSchema logo as long as you have committed to our annual subscription pricing.
We’re actively working on bringing you more granular customization options to better fit in with your product styles — if you have any particular requests, please let us know.
Support for 10M+ row files
Our spreadsheets are running on new infrastructure! Our engineering team has built a new spreadsheet data service in Rust that loads every uploaded file fully in-memory. The spreadsheet files will live on new high-memory servers that make it possible to validate and transform files of up to 4 GB in under 1 second.
Files of up to 1,000,000 rows can be:
- Uploaded in under a minute.
- Validated in under a second.
- Auto-fixed and transformed in under a second.
Excel worksheet selection
Your users will be presented the option to select which worksheet within an Excel workbook they want to upload.
Note: This pane will only appear for users who upload an Excel Workbook with multiple worksheets. CSV uploads and single sheet imports will not trigger this modal.
Mapping UI update
The UI for the Map Column pane has been updated. We’ve added more visual separation to delineate your user’s uploaded columns from the template columns. We’ve also improved the delete UX to make it clearer that columns are deleteable and will not be imported as a result.
These changes do not require any additional configuration from you.
Column descriptions
Write custom descriptions for any column you feel your users could use additional context or instructions. These descriptions can help guide them through the mapping process and during the final validation step. Descriptions will appear in context to each column, both in the Map Column pane and the Review pane.
Expanded validation library (50+ options)
Environment scopes for API keys
We're excited to roll out environment scopes for API keys, which enables those keys to only access embeds in the prescribed environment. This helps customers uplevel the security of their organization by ensuring that their developers' local and staging environments should not have access to customer PII.
CA data residency
AU data residency
Self hosting (Enterprise Feature)
You can now host OneSchema inside of your AWS or Azure cloud environment. The self-hosted model provides full isolation of data in your own cloud, and is best for customers who are handling government data (GovCloud), or other extremely sensitive data with unique contractual / compliance requirements. OneSchema will deploy updates and maintenance to the platform through an IAM user.
SOC 3 Report
Multi-region hosting
Audit logging dashboard & API
GDPR Compliance
HIPAA Compliance
Access management
You can now grant and revoke team member access to OneSchema directly from the developer dashboard.
SOC 2 Type II Compliance
While OneSchema makes the overall process of data migration much faster and far less painful, security and compliance are always top of mind as our customers trust us with their customer's sensitive business data and PII.
We are extremely excited to share the news today that OneSchema has now achieved SOC2 Type II compliance.
Public version of FileFeeds embedding SDK in JS + React
Importer SDK versions 0.6.0 and 0.7.0
A new version of our SDKs for the Importer (0.6.0) is now available for all 4 supported frameworks: JS, React, Vue and Angular.
- Added compatibility with OneSchema FileFeeds packages.
- Exported TypeScript types in Importer React package.
- Dropped TemplateGroups feature.
- Added tslib as direct dependency.
Importer React SDK 0.7.0 update: inline
prop is defaulted to true
When upgrading to this version, if you still need the non-inline behavior, you could set inline={false}
explicitly. Or, you could switch to the inline
rendering (which is more native to React environment) and adjust your style attribute and CSS rules to achieve the desired UI presentation.
Webhook Logs for all webhooks
OneSchema now has logs for all types of supported webhooks:
- Importer Event webhooks
- Importer Metadata webhooks
- FileFeed Event webhooks
FileFeeds Mapping + Transforms experience updates
- Pipelines are now built into FileFeeds and called Transforms
- FileFeeds now respect pushed templates (relying on the latest template version pushed to the Production environment)
- Change history: you can now see the history of saved transforms per FileFeed
- New bulk action & pop-up: “Missing required columns”, clicking on “Add all missing columns” forces all required template columns onto the FileFeed
- “Save transforms” button moved to the top right corner of Transforms
- When uploading a new sample file to a FileFeed, the column mapping modal auto-opens
- Sample file columns mapped to template columns are now highlighted by a green “MAPPED” badge
- Other minor UI/UX updates