AWS Data Exchange for APIs: Automated Data Updates

Automating Third-Party API Updates with AWS Data Exchange

Traditionally, developers integrate machine learning models with external datasets, completing the process after initial inclusion. However, data is rarely unchanging. This necessitates the creation of data pipelines for regular collection, adding to the workload.

During AWS re:Invent in Las Vegas, Amazon Web Services unveiled AWS Data Exchange for APIs. This new service automates updates to evolving third-party APIs, eliminating the need for developers to construct and maintain update mechanisms themselves.

The Benefits of API Integration

According to Alex Casalboni, writing on the AWS blog, APIs allow data scientists to quickly obtain answers to specific queries involving dynamic information, such as current stock prices. This bypasses the requirement for a dedicated data pipeline.

While beneficial for a single API, utilizing multiple APIs introduces complexities. These include challenges related to communication protocols, authentication procedures, and overall API governance.

How AWS Data Exchange for APIs Addresses These Challenges

AWS Data Exchange for APIs is designed to resolve these issues. Casalboni stated, “I’m happy to announce the general availability of AWS Data Exchange for APIs, a new capability that lets you find, subscribe to, and use third-party APIs with a consistent access using AWS SDKs, as well as consistent AWS-native authentication and governance.”

A crucial aspect of this tool is its specific integration with the AWS ecosystem. Applications and data models built on AWS can leverage AWS SDKs and existing authentication/governance tools for automated access and updates to third-party APIs.

Value for Data Providers

Third-party data providers also benefit from this service. Listing their APIs within the AWS Data Exchange catalog increases visibility and accessibility to a wider developer audience.

“As a data provider, you make your API discoverable by millions of AWS customers by listing it in the AWS Data Exchange catalog using an OpenAPI specification and fronting it with an Amazon API Gateway endpoint,” Casalboni clarified.

Available Data Sources

Several third-party providers are already offering data through the exchange, including Variety Business Intelligence, IMDb, and Foursquare. The service is available to both developers and API providers immediately.

Here's a quick overview of the key advantages: