IBM Cloud SQL Query

More customers pick Amazon Redshift than any other cloud data warehouse. Redshift powers analytical workloads for Fortune 500 companies, startups, and everything in between. Companies like Lyft have grown with Redshift from startups to multi-billion dollar enterprises. No other data warehouse makes it as easy to gain new insights from all your data. With Redshift you can query petabytes of structured and semi-structured data across your data warehouse, operational database, and your data lake using standard SQL. Redshift lets you easily save the results of your queries back to your S3 data lake using open formats like Apache Parquet to further analyze from other analytics services like Amazon EMR, Amazon Athena, and Amazon SageMaker. Redshift is the world’s fastest cloud data warehouse and gets faster every year. For performance intensive workloads you can use the new RA3 instances to get up to 3x the performance of any cloud data warehouse.

Apache DataFusion is an extensible, high-performance query engine written in Rust that utilizes Apache Arrow as its in-memory format. Designed for developers building data-centric systems such as databases, data frames, machine learning, and streaming applications, DataFusion offers SQL and DataFrame APIs, a vectorized, multi-threaded, streaming execution engine, and support for partitioned data sources. It natively supports formats like CSV, Parquet, JSON, and Avro, and allows for seamless integration with object stores including AWS S3, Azure Blob Storage, and Google Cloud Storage. The engine features a comprehensive query planner, a state-of-the-art optimizer with capabilities like expression coercion and simplification, projection and filter pushdown, sort and distribution-aware optimizations, and automatic join reordering. DataFusion is highly customizable, enabling the addition of user-defined scalar, aggregate, and window functions, custom data sources, query languages, etc.